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Avant-Propos 


1  ,'OTAN  avail  organise  dcs  1965  un  Cours  Avance  d'enc  quaiite  exemplaire  sur  I'Evaluation  des  Sytemes  d'lnformation,  cours 
qiii  rcunissait  a  la  Haye  les  meilleurs  specialistcs  en  sersHee  de  rinformation. 

La  notion  de  systeme  est  importante  parce  qu’elle  peimet  toutes  les  evolutions  souhaitables  pour  une  meilleure  satisfaction  des 
bes'  ins  de  I'utilisateur.  Elle  permet  a  la  cybernelique  de  laisser  sa  place  a  rhumain.  Elle  montre  quo  I'efficacite  est  liee  a  la 
mesure  d’un  certain  nombre  de  parametres  qu'il  faut  savoir  identifier  et  maitriser. 

Au  ..«in  des  pays  de  I’OTAN  les  grands  centres  ou  services  d'lnformation  ont  une  part  importante  de  responsabilite  dans  la 
const  'jction  et  revolution  des  systemes  nationaux.  Cest  pourquoi  la  Commission  de  I'lnformation  Technique  de  I'AGARD  a 
organise  en  septtmbre  i988  une  Sene  de  Conferences  sur  le  theme  de  “revaluation  des  Centres  el  Services  d'lnformation”  au 
Luxeir  bourg,  en  Grece  et  au  Portugi  I.  Les  lextes  de  ces  conferences  ont  etc  rassembles  dans  la  serie  de  conferences  no.  160. 

La  Commission  a  alors  demande  au  directeur  de  la  sene  de  conferences  de  combiner  le  contemi  de  ces  presentations  et 
I'experien: .  qu'il  a  lui  meme  acquise  a  I'occasioii  .le  sa  participation  a  dcs  actions  d'evaluation  de  centres  et  services 
d'informalion,  dc  fajon  a  pieparer  un  manuel  tenant  lieu  de  synth^c:  e'est  le  volume  presenie  ici. 

A  I'heure  o.:  il  apparail  clairement  que  "I'investissement  immalcriel”  dans  I'information  est  essentiel  si  I’on  veut  resler 
competitif,  le  manuel  permettra  a  tous  ceux  qui  ont  une  responsabilite  dans  la  gestion  et  le  transfer!  de  I'information  de  liter  le 
meilleur  parti  des  ressourccs  et  moyens  a  leui  disposition,  d'etre  “proaclifs”  plulot  que  “reactifs"  en  mellant  en  place  les 
systemes  de  mesure,  les  tableaux  de  boros,  qui  faciliteront  les  prises  de  decision. 

Je  remercie  les  auteurs  de  ce  manuel,  Jose-Marie  Cnffiths  et  Donald  W.King,  qji  ont  me  scmble-t-il  parfailement  rcussi  a 
regrouper  e!  a  organiser  dans  cet  ouvrage  les  connaissances  dcs  meilleurs  experts  du  domaine  et  done  a  facihter  la  mise  en  place 
de  CCS  systemes  dc  mesure  et  d’evaluation. 

Ce  manuel  devrait  faire  I’objet  d'une  large  diffusion  aupres  des  personnels  techniques  et  administralifs  qui  unt  ou  sont  appelcs  a 
avoir  un  role  important  dans  la  gcslion  et  le  transfer!  de  I'information. 


ALBERT YANEZ 
President  dc  la  Commn  sion  dc 
I’information  Technique  Je  I’AGARD 
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Foreword 


In  rJ65,  NATO  organised  an  advanced  course  on  the  evaluation  of  information  systems.  The  course  was  of  an  exemplary 
standard  and  brought  together  at  the  Hague  the  leading  specialists  in  the  field  of  information  services. 

The  cotcept  of  systen  L  mportant,  as  it  provides  the  necessary  flexibility  for  a  type  of  development  more  in  line  with  user 
requiremen's.  It  enables  cybernetics  to  give  way  to  the  human  being.  It  demonstrates  that  efficiency  is  linked  to  the  ability  to 
measure  a  number  of  pat  arnetors  which  the  user  must  know  how  to  identify  and  master. 

In  the  NATO  nations  the  maior  information  centres  or  services  have  a  large  share  of  responsibility  for  the  design  and 
development  of  national  systems  For  the.se  rca.sons.  'he  Technical  Information  Panel  of  AGARD  held  a  Lecture  Series  on  “The 
evaluation  of  information  centres  and  services”  .1  Luxembourg,  Greece  and  Portugal.  The  texts  of  the  lectures  are  contained  in 
Lecture  Senes  N0.I6O. 

The  Panel  then  commissioned  the  Director  of  the  Lecture  Senes  to  combine  the  material  presented  there  with  the  experience  ne 
had  gained  in  many  projects  involving  the  evaluation  of  information  centres  and  .services,  and  prepare  a  unified  manual  (this 
AGARDograph). 

At  a  lime  when  it  seems  clear  that  “grey  matter  investment”  in  information  is  vital  it  we  are  to  remain  competitive,  this  manual 
will  enable  all  those  responsible  for  the  management  and  transfer  of  information  to  get  the  best  out  of  the  resources  and  facilities 
at  their  disposal,  and  to  be  "proactive"  rather  than  "reactive",  by  selling  up  measurement  systems  which  act  as  instrument  panels 
for  those  responsible  for  making  the  decisions. 

I  would  like  to  thank  the  authors  of  this  manual,  Jose-Marie  Griffiths  and  Donald  W  King,  who,  in  my  opinion,  have  successfully 
brought  together  and  structured  in  one  volume  the  knowledge  held  by  the  leading  experts  in  the  field,  and  by  so  doing  have 
facilitated  the  implementation  of  these  measurement  and  evaluation  systems. 

This  manual  should  be  widely  disseminated  among  the  administrative  and  technical  personnel  who  have  or  will  have  an 
important  role  to  play  in  the  management  and  transfer  of  information 


ALBERT YANEZ 
Chairman 

Technical  Information  Panel  of  AGARD 


Preface 


This  manual  on  “The  Evaluation  of  Information  Centers  and  Services"  grew  out  of  AGARD  Lecture  Scries  No  160  entitled 
"Evaluating  the  Effectiveness  of  Information  Centres  and  Services"  The  Lecture  Series  was  given  by. 

Dr.  Jose-Mane  Griffiths,  King  Research,  Inc. 

Donald  W.  King,  King  Research,  Inc. 

John  Martyn,  Polytechnic  of  Central  London 

Professor  Jack  Meadows,  Dean,  Dept,  of  Library  Infoi  matron  Studies 

Dr.  David  Penniman.  AT&T  Bell  I  .aboratories 

in  September  1988  in  the  host  countries  of  Luxembourg,  Greece,  and  Portugal 
Readers  of  this  manual  are  particularly  recommended  to  read  the  papers'- 

Measures,  Methods  and  Models  Employed  in  Evaluating  the  Effectiveness  of  Information  Centres  and  Services,  by  A  J 
Meadows 

European  Examples  of  Evaluating  the  Effectiveness  of  Information  Centres  and  Services,  by  J.  Martyn 
Evaluating  for  Information  Centre  Planning,  by  W.  E.  Penniman. 

This  manual  deals  largely  with  extensive  evaluations  done  by  King  Research  over  the  pas'  fifteen  years.  Tlie  genesis  of  the 
approach  to  evaluation  of  information  centers  and  services  discussed  here  began  with  a  National  Science  Foundation  study 
published  as  a  book-  in  1971.  Since  'hat  time  the  approach  has  been  modified  somewhat  and  extended  considerably  as  a  result 
of  nearly  300  projects  involving  planning,  evaluation  and  dc.sign  of  information  centers  and  services.  We  emphasize  that  the 
general  approach  presented  here  (including  suggested  measures,  models  and  methods)  are  not  the  only  ones  that  one  might  use 
Rather,  the  approach  in  this  manual  is  a  result  of  the  particular  knowledge  we  have  gained  and  it  rcfiects  what  we  have  found  to 
be  applicable  and  useful  to  funders,  managers,  information  center  staff  and  information  service  users.  For  other  approaches  to 
evaluation,  we  refer  the  reader  to  the  bibliography  at  the  end  of  the  manual  and,  m  particular,  the  works  of  Buckland,  Hayes, 
Kantor,  Martyn,  Meadows,  and  Penniman. 

In  this  manual  we  emphasize  evaluation  measures,  models,  and  methods  and  we  usually  present  actual  data  and  results  from 
studies  performed  by  King  Research  A  companion  book,  "Special  Libraries  and  Information  Services  —  Increasing  the 
Intormation  Edge”’  represents  an  overview  of  the  results  of  these  studies.  Also,  -we  prepared  “Keys  to  Success  Performance 
Indicators  for  Public  Libraries”^  that  provides  an  additional  perspective  for  using  evaluation  measures  and  indicators 


Josc-Marie  Griffiths,  Ph  D. 
Donald  iV.  King 
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Chapter  1 

The  Role  of  Evaluaiion 


1.1  Background 

Information  systems  evaluation  has  been  extensively 
discussed  in  the  literature  over  the  past  thirty  years  and  there 
have  been  many  exemplary  evaluations  performed  over  the 
years.  However,  we  believe  that  evaluation  of  information 
centers  and  services  is  likely  to  take  on  new  significance  in 
the  1990's.  Institutions  of  all  kinds  worldwide  arc 
undergoing  intensive  scrutiny  in  terms  of  their  performance 
and  effectiveness,  we  believe  properly  so.  It  is  no  longer 
accepted  as  matter  of  fact  that  information  centers  are 
necessary  or  that  all  services  are  needed.  Information 
centers  arc  undergoing  increasingly  strong  competition  for 
funds  within  their  companies,  educational  institutions, 
government  agencies  or  other  organizations.  We  believe  that 
such  competition  will  become  even  more  intense  during  the 
199()'s.  Unfortunately,  many  information  centers  are  not 
well  prepared  to  meet  this  competition.  They  typically  do 
not  measure  or  keep  data  that  arc  useful  for  making  a 
compelling  case  for  themselves  in  a  highly  competitive 
environment. 

Most  iiifoimation  centers  that  we  have  dealt  with  maintain 
their  management  information  or  data  in  terms  of  budgeted 
Items  that  reflect  resources  such  as  staff,  collections  (stock), 
equipment  and  systems,  and  facilities.  Sometimes 
information  center  managers  allocate  budgets  to  specific 
services,  but  rarely  do  they  measure  the  performance  of 
those  services  in  terms  ot  productivity  or  output  quality, 
timeliness,  etc.  It  is  even  rarer  for  information  center 
managers  to  establish  the  effectiveness  of  their  services  in 
terms  of  user  satisfaction,  the  extent  to  which  services  are 
used  and  the  consequences  of  use  of  the  services  in  terms  of 
the  purposes  for  which  services  are  used  and  how  services 
affect  users'  work.  We  believe  that  in  the  future,  budgets  for 
information  centers  and  services  will  be  considered  an 
investment  and  return  on  that  investment  must  be 
considered  in  terms  of  how  effective  services  are  for  meeting 
the  organizations’  mission,  goals  and  objectives. 

Throughout  the  Manual  are  several  recurring  evaluation 
themes: 

•  Evaluation  must  have  a  purpose  it  must  not  be  an  end  in 
Itself.  The  purpose  of  evaluation  arises  out  of  a  need  to 
“set  a  value  on”  an  information  center  and  services. 
Lord  Kelvin  has  said  that  “..when  you  can  measure 
what  you  are  speaking  about,  and  express  it  in 
numbers,  you  know  something  about  it;  but  when  you 
cannot  measure  it,  when  you  cannot  express  it  in 
numbers,  your  knowledge  is  of  a  meager  and 
unsatisfactory  kind...”’. 

•  Without  the  potential for  some  action,  there  is  no  need  to 
evaluate.  Whether  evaluation  is  used  for  budgeting, 
planning,  administration,  design  or  other  functions,  it 
should  lead  to  decision-making,  which  is  a  prelude  to 
action. 

•  Evaluation  must  be  more  than  merely  descriptive; 


rather,  evaluation  must  take  into  account  relationships 
between  operational  performance  and  effects  on  the 
users  and  organizations  being  served.  In  addition  to 
merely  measuring  costs  and  amounts  of  services 
provided,  it  is  important  to  be  able  to  relate  costs  to  a 
combination  of  service  quantities,  quality,  timeliness, 
etc.  and,  in  turn,  to  relate  quality  and  timeliness  to  user 
satisfaction,  extent  of  use  and  consequences  of  use. 

•  Evaluaiion  should  be  a  communication  tool  where 
feedback  to  and  from  staff  and  users  is  unimpeded  .11 
evaluation  should  be  addressed  to,  made  known  oy 
and  used  by  information  center  funders,  managers, 
staff  and  users.  We  strongly  advocate  a  participatory 
approach  to  evaluation.  Information  center  staff  must 
know  the  objectives  and  possible  outcomes  of 
evaluation.  They  can  contribute  substantially  to 
evaluation  through  their  intimate  knowledge  of 
operations  and  users.  They  can  also  perform  their 
work  better  as  a  result  of  knowledge  gained  through 
evaluation,  whether  it  be  operational  or  user-related 
knowledge.  We  have  found  that  users  also  can  benefit 
from  the  knowledge  gained  through  evaluation  — 
sometimes  regarding  their  mterfaee  with  serviee  staff, 
use  of  services  or  how  information  affeets  their  work. 

•  Evaluation  should  not  be  spomdic  in  nature,  but  rather 
should  he  considered  as  an  ongoing  management  tool 
supported  by  an  ongoing  system  of  measures  or 
management  information  system  Nearly  all  evaluation 
reported  m  the  literature  involve  one-time  studies 
However,  ongoing  evaluation  can  be  relatively 
inexpensive,  because  observation  and  analysis  costs 
arc  spread  over  time.  Furthermore,  a  one-time 
evaluation  provides  a  baseline  measurement  only.  It  is 
the  ongoing  evaluation  subsequent  to  aetion  that 
provides  important  information  on  the  effects  of 
action. 

•  Ongoing  evaluation  should  provide  a  means  for 
cominual  monitoring,  diagnosis  and  change.  Such 
ongoing  evaluation  makes  managers  proactive  rather 
than  reactive.  By  observing  trends  in  critical  measures 
and  indicators,  managers  can  set  measurable 
objectives  which  help  fulfill  meaningful  goals  and 
missions  of  the  center  and  services.  A  system  of 
measures  will  provide  diagnostic  tools  for  determining 
why  objectives  are  not  achieved  and  what  the 
consequences  of  not  meeting  the  objectives  are  to  the 
parent  organization. 

•  Ongoing  evaluation  should  he  dynamic  in  nature,  in  that 
evaluaiion  measures,  models  and  methods  must  reflect 
new  knowledge  and  changes  in  the  operational,  user  and 
organizational  environment.  Changes  in  information 
center  operations  and  services  will  of  necessity  require 
different  evaluaiion  measures,  iiiudeis  and  ilieihuds. 
Furthermore,  knowledge  gained  through  continued 
observation  and  analysis  will  dictate  modification  to 
evaluation  procedures,  measures,  models  and 
methods. 
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Evaluation  costs  money  and  uses  resources  that  might  he 
applied  for  competing  functions  m  the  center.  The 
investment  is  m  information  that  should  lead  to  better 
decision-making  and  ultimate  savings  to  the  information 
center.  The  favorable  consequences  of  evaluation  and 
ongoing  system  of  measures  must  exceed  the  costs. 
Unfortunately  there  is  little  documentation,  except  through 
anecdotes  to  demonstrate  this.  On  the  other  hand, 
circumstances  m  the  future  may  impose  the  need  for 
continued  evaluation  and  it  behooves  information  center 
management  to  be  prepared  for  this.  We  are  aware  of  no 
guidelines  regarding  how  much  evaluation  should  cost. 
However,  we  believe  that  ongoing  evaluation  and  svjiems  of 
measures  should  cost  in  the  neighborhood  of  iwo  to  five 
percent  of  budget.  When  a  large-scale  change  is  anticipated 
such  as  purchase  or  development  o^  a  major  automated 
system  or  a  move  to  new  facilities,  approximately  five 
percent  of  that  budget  should  be  set  aside  for  special- 
purpose  evaluation  tied  to  the  ongoing  evaluation.  One  can 
cite  hundreds  of  examples  where,  without  proper 
evaluation,  the  cost  of  mistakes  made  in  purchase  of  new 
systems  cost  the  information  center  literally  ten  to  20  times 
the  recommended  amount  of  five  percent  investment  in 
evaluation. 

1.2.  The  Role  of  Evaluation 

We  emphasize  throughout  the  Manual  the  many  roles  that 
evaluation  plays  in  information  centers.  We  also  stress  that 
evaluation  should  not  be  a  one-time  process  but  rather 
should  provide  a  continual  system  of  measures  and 
management  information.  In  particular,  evaluation  and  a 
system  of  measures  should  assist  in  the  following  functions, 

•  Planning 

•  Resource  management 

•  Operational  function  and  service  management 

•  Promotion,  marketing  and  public  relations 

The  role  of  each  of  these  functions  is  discussed  below. 
Evaluation  During  Planning 

Basically,  the  planning  process  is  a  structured  framework  for 
continuous  problem  solving  based  on  a  combination  of 
objective  and  subjective  information.  The  role  of  evaluation 
in  this  process  is  to  provide  the  basic  information  for 
designing  and  redesigning  information  center  operational 
functions,  services,  or  systems.  A  planning  manual  was 
developed  by  Palmour  and  colleagues  at  King  Research 
and  evaluation  aspects  from  that  manual  arc  described 
below. 

Planning  entails  making  decisions  based  on  predicted 
effects  of  alternative  actions.  Deci.sions  are  also  made  on  the 
basis  of  results  of  past  decisions.  This  is  the  control  function 
of  both  planning  and  evaluation.  Techniques  most  often 
used  in  evaluation  during  the  planning  process  include 
observational  studies,  surveys,  and  descriptive  models  of  the 
system.  For  example,  once  a  planning  group  or  committee 
has  developed  several  strategies  for  possible 
implementation,  it  is  necessary  to  evaluate  the  strategies  to 
determine  the  best  methods  of  reaching  the  desired  ends. 
Techniques  used  for  evaluating  proposed  strategies  (i.e.,  for 
evalua.’ng  alternative  actions  for  accomplishing  the 
previojsly  determined  objectives)  include  cost  finding, 
assessi  aent  of  center  records,  surveys  and  experimentation. 


Evaluation  provides  a  guide  for  considering  the  probable 
impact  of  a  strategy  upon  each  objective.  The  question  is 
essentially  “How  much  better  can  the  center,  system  or  new 
technology  accomplish  each  of  its  objectives  if  the  strategy  is 
adopted  and  implemented?"  Because  a  strategy  designed  to 
further  one  objective  may  affect  another,  for  better  or  worse, 
each  strategy  has  to  be  evaluated  against  each  objective.  This 
requires  some  means  of  anticipating  or  forecasting  the 
incremental  gain  or  loss  from  what  is  currently  being 
accomplished.  Tins  may  be  a  subjective  exerci.se,  because  it 
relies  on  the  expectations  of  the  effects  of  strategics  on 
objectives;  but  it  is  absolutely  necessary  to  evaluate 
strategics  for  possible  implementation.  There  is  also  the 
experimental  approach,  whereby  strategies  are  tested  by 
limited  application  in  only  one  part  of  the  system. 

Planning,  like  the  evaluation  process  it  includes,  is  ongoing. 
A  long-range  plan  is  developed  as  pan  of  the  initial  cycle  of 
the  planning  process.  It  includes  the  monitoring  and 
evaluation  of  the  center's  plan  itself  and  evaluation  of  the 
continued  relevance  of  current  center  operations,  services, 
and  products  to  the  needs  of  the  users  to  be  served. 

Planning  involves  setting  an  information  center's  mission, 
goals,  measurable  objectives,  pnonties  and  strategies  for 
achieving  change.  These  aspects  of  planning  and 
management  are  defined  as  follows: 

•  Mission  should  be  an  overall  statement  of  the 
information  center's  role  in  meeting  its  parent 
organization's  mission,  goals  and  objectives.  It  should 
reflect  the  center's  philosophy  in  providing  service. 

•  Goals  are  broad  statements  of  desired  ends  such  as  an 
increase  in  the  number  of  users,  an  increase  in  the  use 
of  services,  automation  of  certain  functions, 
development  of  new  services,  etc. 

•  Objectives  arc  specific  targets  within  the  goals,  and 
there  may  be  several  objectives  for  each  goal.  The 
objectives  should  be  measurable,  if  possible 

•  Priorities  should  be  set  among  objectives  to  help  make 
decisions  on  actions  under  various  funding  and  other 
resource  limilation/contingencies. 

•  Strategies  for  change  are  the '  .ctions  which  can  be  taken 
to  achieve  the  goals  and  objectives 

In  our  framework  for  evaluation  we  stress  several 
perspectives  for  evaluation  including  the  information  center 
(management  and  staff),  users  and  the  user's  organizations 
(often  the  funder  of  the  center)  The  mission  statement 
should  include  some  language  from  the  organization's 
perspective,  some  goals  must  include  meeting  users' 
information  needs  and  requirements,  and  some  objectives 
can  include  pcrfotmance  and  effectiveness  measures.  We 
show  that  we  can  link  information  service  performance  to 
users’  performance.  Thus,  one  can  conceptually  also  link  the 
information  center's  performance  to  the  objectives,  goals 
and  mission  of  the  parent  organization. 

Evaluation  and  information  requirements  for  the  planning 
process  (for  public  libraries)  were  developed  as  shown  in 
Figure  1  The  structure  holds  for  information  centers  as  well. 

A  useful  application  of  evaluation  is  in  planning  and 
designing  systems  and  automation  in  mforinatiuii  eciiieis.  It 
is  essential  in  planning,  feasibility  analysis,  design, 
development  and  operations.  Evaluation  involving  new 
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Fig.1  Information  requirements  for  the  planning  process 


technology  must  be  an  integral  part  of  the  entire  system  life 
cycle.  Information  systems  have  life  cycles  that  consist  of 
overlapping,  interconnected,  and  alternative  phases- 
planning  and  feasibility  analysis,  design,  implementation, 
operation,  and  planning  again.  As  shown  in  Figure  2, 
evaluation  can  have  a  role  during  each  phase.  In  general, 
evaluation  methods  used  during  the  planning  and  operating 
phases  differ  somewhat  from  those  used  during  the  design 
and  implementation  stages.  The  first  phases  make  more 
t  ttensive  use  of  simulation  and  experimentation  techniques. 


while  the  latter  phases  provide  more  opportunity  for 
observational  and  descriptive  techniques.  The 
implementation  phase  uses  both  types  of  techniques 

Evaluation  research  provides  the  basic  information  for 
designing  and  redesignmg  systems.  In  addition,  c-valuation 
studies  have  as  one  purpose  the  assessment  of  designs  prior 
to  implementation.  In  the  design  process,  the  decision¬ 
maker  must  begin  to  consider  multiple  alternatives  under 
multiple  conditions.  The  selection  from  alternative  systems 
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is  rarely  simple  with  one  alternative  clearly  dominating  all 
others.  Usually,  one  system  or  alternative  technology 
appears  to  be  superior  wi.h  respect  to  one  objective  but  not 
to  another.  The  evaluator’s  function  is  to  provide  the 
decision-maker  with  an  explicit  and  rational  analysis. 
Consequently,  evaluation  of  design  alternatives  .often 
involves  the  use  of  models.  Depending  on  the  alternative,  a 
model  may  involve  only  verbal  statements  of  cause  and 
effect;  or  mathematical  models  can  be  developed  to  provide 
necessary  data.  Computer  simulation  may  be  also  used. 

In  some  instances,  research  or  feasibility  analysis  precedes 
design.  When  research  is  performed,  evaluation  plays  an 
important  role  in  observing,  describing,  and  simulating  the 
information  environment  under  different  conditions.  Tbese 
evaluation  techniques  provide  the  opportunity  to  consider 
the  information  environment  under  different  conditions  and 
perhaps  anticipate  potential  problem  areas  before  the  actual 
system  IS  operational.  Experiments  conducted  as  part  of  the 
evaluation  procedure  in  such  a  research  context  can  be 
performed  to  understand  more  of  the  phenomenon  of  user 
satisfaction  by  measumig  as  many  of  the  different  aspects  of 
it  as  possible.  The  increased  knowledge  of  user  behavior  can 
be  used  to  adjust  the  conditions  surrounding  computer- 
based  or  electronic  services  accurately  in  order  to  increase 
user  satisfaction. 


SYSTSl  L!F£  CYCLE 


Fig.2  Evaluation  as  a  part  of  a  system  life  cycle 


Once  a  plan  has  been  developed  and  the  system  designed, 
the  next  step  is  its  implementation.  Evaluation  is  performed 
during  this  phase  to  check  the  match  between  the 
implementation  and  program  expectations.  Evaluation 
study  results  may  suggest  needed  adjustments  before  the 
system  becomes  operational.  The  principal  objectives  of  this 
evaluation  are  (1)  to  predict  the  performance  of  the  system 
once  it  is  made  operational;  (2)  to  reveal  specific  needs,  if 
any,  for  modifying  and  correcting  the  system  before  changes 
become  too  costly;  and  (3)  to  perform  a  preliminary 
investigation  of  operational  strategies. 


As  part  of  the  planning  process,  decisionmaking  is  based  on 
the  predicted  effects  of  alternative  actions.  In  the 
operational  stage  of  the  information  system,  decisions  are 
also  being  made  on  the  basis  of  the  results  of  past  decisions. 
There  arc  numerous  examples  of  the  evaluation  of 
operational  systems  —  evaluation  is  undertaken  routinely  in 
connection  with  quality  control  and  in  connection  with  the 
ongoing,  longrange  planning  process  that  is  part  of  most 
operational  systems. 

Evaluation  studies  of  operational  systems  have  one  or  more 
of  the  following  purposes:  ( 1 )  to  discover  whether  and  how 
well  objectives  are  being  fulfilled,  (2)  to  determine  the 
rca,sons  for  specific  successes  and  failures,  and  (3)  to 
uncover  the  principles  underlying  a  successful  program. 
Questions  fur  evaluation  at  the  operational  stage  include: 
How  good  is  the  technology?  What  effects  docs  it  have? 
Does  it  work  as  expected?  The  quality  of  the  results 
(expected  or  realized)  is  weighted  against  the  resources 
required  to  implement. 

Information  upon  which  to  base  the  evaluation  of  current 
services  is  gathered  from  various  sources’  currently 
collected  statistics,  measures  of  system  performance  or 
effectiveness,  and  surveys.  For  example,  the  methods  used  in 
designing  an  information  retncval  system  for  a  group  of 
users  (eg.,  in-dcpth  interviewing  and  user  profiles)  can  be 
used  to  collect  new  data  that,  when  compared  with  earlier 
data,  make  it  po.ssiblc  to  evaluate  the  success  of  the  system  in 
attracting  its  target  population 

After  the  equipment  has  been  installed  and  operational  for 
six  months  to  a  year,  a  post-implementation  evaluation 
might  be  performed.  This  evaluation  will  determine  whether 
or  not  the  new  system  meets  the  objectives  stated  during  the 
planning,  analysis  and  design  activities.  The  evaluation 
should  be  as  objective  as  possible.  Consequently. 
con.sultants  arc  often  asked  to  perform  such  evaluation.  The 
evaluation  results  should  feed  back  directly  into  the 
planning  process,  initiating  a  whole  new  cycle. 

Evaliimion  for  Resource  Mtinugemcm 

Wc  indicate  that  information  center  resources  include  staff, 
collections,  equipment  and  systems,  facilities,  supplies  and 
so  on.  Most  information  center  budgets  are  based  on  such 
resources  in  an  optional  way.  This  is  as  it  should  be  because 
an  important  part  of  center  management  is  to  allocate 
resources.  Part  of  a  resource  allocation  involves  the 
allocation  of  funds  among  staff,  systems,  facilities,  etc. 
Another  part  involves  allocation  within  resources.  For 
example,  personnel  management  includes:  (1) 
determination  of  information  center  staffing  patterns  (i.e., 
how  many  of  the  staff  should  be  professional, 
paraprofessional  and  support  and  at  what  level  for  each)  and 
(2)  penodic  review  of  information  center  units  and 
individual  staff  members.  System  and  equipment 
management  includes  determination  of  system  needs  and 
review  of  adherence  of  performance  to  contract 
specifications.  Facilities  management  involves  determining 
the  location,  size,  ambience,  and  amount  of  research  space, 
reading  space,  staff  space  requirements,  etc.  that  arc 
necessary;  including  adjustments  for  growth  and  other 
change.s. 

We  suggest  two  basic  types  of  measures  for  evaluating 
resources. The  first  measure  is  of  input  cost,  which  is  defined 
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as  the  application  of  resources  to  various  operational 
functions  and  services.  Input  of  resources  can  also  be 
described  in  terms  of  the  attributes  of  tesources  such  as  staff 
competence,  experience,  etc.,  or  equipment  speed, 
reliability,  etc.,  applied  to  the  various  functions.  Ihe  second 
measure  is  of  the  output  of  resources  in  terms  of  quantities 
and  attributes.  Performance  of  resources,  then,  includes 
these  measures  and  relationships  between  them  (e.g., 
productivity).  One  can  apply  the  results  of  such  evaluation 
to  financial  management  including  annual  budgeting, 
preparation  of  ad  hoc  modifications  to  the  budget,  and 
monitoring  adherence  to  the  budget. 

Ewlaation  for  Operational  Function  and  Servie- 
Management 

Wfile  most  information  centers  maintain  detailed 
acco.'nting  records  related  to  resources,  no:  nearly  as  many 
maintain  data  concerning  operational  functions  and 
services.  Yet  functions  and  services  are  the  essence  of 
information  centers  and  .should  be  routinely  evaluated.  The 
data  that  are  available  usually  concern  output  quantities 
such  as  the  number  of  items  acquired,  number  of  online 
bibliographic  searches  performed  and  so  on.  Howe’.er, 
performance  evaluation  must  also  include  input  costs  and 
the  relationships  of  input  costs  (and  resource  attributes)  to 
output  quantities  (and  attributes)  such  as  quality,  timeliness, 
etc.  Input  costs  involve  summing  all  resources  allocated  to 
an  operational  function  or  service,  using  a  common  unit  of 
measure  (i.e.,  dollars).  In  order  to  do  this,  the  amount  of 
resources  such  as  staff  time  must  be  allocated  to  specific 
functions  and  services  using  cost  finding  methods. 

Throughout  the  Manuai  we  describe  information  centers  in 
terms  of  three  kinds  of  functions:  user-related  functions 
(e.g.,  reference,  access  to  facilities  and  systems,  access  to 
collection  and  materials,  etc.),  operational  functions  (e  g., 
acquisitions,  technical  processing,  etc.)  and  support 
functions  (e.g.,  management,  administration,  personnel, 
finance,  etc.).  Functions  are  grouped  in  this  way  because 
each  group  has  unique  impiications  for  evaluation 
objectives,  measures,  and  methods.  For  example, 
effectiveness  of  user-related  functions  or  services  is 
measured  in  terms  of  how  output  affects  users;  effectiveness 
of  operational  functions  is  how  output  affects  user-related 
services;  and  effectiveness  of  support  functions  is  how  well 
the  center  operates. 

Evaluating  Promotion,  Marketing  and  Public  Relations 
Functions 

Even  though  promotion  of  an  infoimation  center,  marketing 
Its  services  and  public  relations  might  properly  be 
considered  support  functions,  we  separate  them  out  because 
few  information  centers  that  we  have  evaluated  perform 
these  activities  well  if  at  all.  At  minimum,  evaluation  should 
include  a  determination  of  users’  knowledge  and  awareness 
of  information  centers  and  their  services.  On  occasion  we 
have  had  center  managers  or  staff  indicate  that  they  do  not 
promote  services  because  promotion  will  create  additional 
work  and  their  budget  will  not  support  this.  We  have  tried  to 
set  forth  an  evaluation  framework  that  will  convince  the 
center’s  parent  organization  or  funders  that  the  information 
center  should  be  used  as  much  as  possible  because  use  of  its 
services  will  result  in  lower  organization  costs  and  better 
operations. 


1.3  An  Approach  to  Evaluation 

Evaluation  as  disclosed  in  this  Manuai  can  be  performed 
from  several  perspectives.  The  most  important  perspective 
is  that  of  the  information  center  with  its  operational 
functions  and  user-related  services.  However,  even  though 
evaluation  may  focus  on  a  particular  information  center 
resource  such  as  staff,  automated  system  etc.  or  on  a  service 
such  as  online  bibliographic  search,  the  evaluation  should  be 
done  in  a  way  that  the  other  perspectives  are  considered  as 
well.  In  other  words,  evaluation  should  go  beyond  the 
examining  of  performance  of  resources  and  services  to  also 
estab'tching  the  effectiveness  or  consequences  of  them, 
inus,  fir  example  if  an  evaluation  involves  online 
bibliogriphic  searching  one  can  establish  service 
perfo  mance  in  such  terms  as  cost,  quantities  oroduced, 
quality,  timeliness,  availability,  cost  per  search,  and  so  on. 
However,  one  can  also  determine  the  effectiveness  of 
searching  in  terms  of  user  satisfaction,  amount  of  use, 
purpose  of  use  and  consequences  of  use  on  the  users'  work 
(i.e.,  users’  productivity,  quality  and  timeliness  of  work,  etc.). 
Funders  and  users  are  interested  in  effectiveness  as  well  as 
in  the  performance  of  operations  and  services,  and 
oftentimes  more  so. 

In  Chapter  2  we  set  forth  a  framework  for  performing 
evaluation  of  information  centers  and  services.  In  the 
framework  we  suggest  a  hierarchical  description  of 
information  centers,  including  functions,  services  and 
products,  acti'ities,  resources  and  resource  characteristics, 
information  centers  perform  three  basic  types  of  functions: 
user-related  functions  (e.g.,  reference,  user  training  and 
access  to  materials,  equipment  and  facilities),  operational 
functions  that  support  user  services  (e.g.,  acquisitions, 
document  processing,  storage,  etc)  and  support  functions 
(c.g.,  management,  finance,  personnel,  etc,).  Each  function 
can  involve  several  services  and  products.  For  example,  the 
reference  function  might  include  referral,  question 
answering,  reference  searching,  selective  dissemination  of 
information  (SDl  services),  etc.  For  each  service  a  number 
of  activities  (or  t.Tsks)  must  be  performed  in  order  to 
perform  the  service  or  produce  a  product.  For  example, 
activities  for  the  service  of  reference  searching  include 
interviewing  users,  developing  search  strategies,  deciding  on 
search  methods,  actual  search,  screening  output,  presenting 
results  to  users,  etc.  In  order  to  perform  activities  there  must 
be  resources  such  as  staff,  equipment,  facilities, 
communication  vendors,  supplies,  etc.  Activities  describe 
what  IS  done  and  resources  are  who  or  what  is  necessary  to 
perform  activities,  Finaily,  the  .owest  level  in  this  hierarchy 
includes  characteristics  or  attributes  of  resources  such  as 
staff  competence  (i.e.,  knowledge,  skills  and  attitudes). 
experir.ice,  education  or  equipment  quality,  reliability,  etc. 
One  can  address  evaluation  at  any  or  all  of  the  levels  in  the 
hierarchy. 

At  each  level  one  can  measure  input  costs  and  output 
quantities  and  attributes  (e.g.,  quality,  timeliness,  etc.).  Costs 
can  be  aggregated  or  summed  to  obtain  costs  at  each 
subsequent  level  in  tlie  hierarchy  up  to  the  point  that  the  cost 
of  the  entire  information  center  is  measured.  Thus,  one  can 
“set  a  value”  on  input  costs  and  output  at  all  levels,  and  also 
establish  relationships  of  input  and  output  both  within  a 
level  (e.g,  productivity  or  cost  per  unit)  and  among  levels.  As 
an  example  of  the  latter,  one  should  1^  able  to  establish  the 
relationship  of  a  searcher’s  competency  to  (1)  input  cost 
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(salary),  (2)  output  quantities  (nu-nber  of  searches 
performed  in  a  year)  and  quality,  and  (3)  productivity 
(quantity  divided  by  cost  at  different  levels  of  quality). 
Examples  of  these  relationships  are  discussed  throughout 
the  Manual.  In  Chapter  2  —  A  Framework  for  Evaluating 
Information  Centers,  the  framework  also  shows  that  users’ 
work  involves  a  hierarchy  of  functions,  activities  and 
resources  (including  information  as  one  resource).  One  can 
also  show  relationships  between  the  use  of  information  or 
information  service  performance  on  users'  input  costs, 
output  (quantities,  quality,  etc.)  and  productivity.  This 
provides  a  powerful  evaluation  and  information  tool  for 
information  center  managers. 

We  present  a  system  of  measures  that  one  can  use  iit 
evaluation.  Included  are  input  cost  and  resource  attribute 
(e.g.,  staff  characteristics)  measures;  measures  of  output 
quantities  and  attributes  (eg.,  quality,  timeliness,  etc); 
effectiveness  measures  (e.g.,  user  satisfaction,  amount  of 
use,  purpose  of  use,  consequences  of  use,  etc.);  and  domain 
measures  (e.g,  numbers  of  persons  served,  their  information 
needs,  etc.).  Measures  by  themselves  convey  little  or  no 
meaning.  Thus,  we  also  present  derived  measures  or 
indicators  that  yield  greater  meaning  and  usefulness.  In 
Chapter  3  Evaluation  Measures,  Models  and  Methods, 
adapted  from  Keys  'o  Success'*,  we  show  that  data  can  be 
presented  in  several  forms,  each  of  which  conveys  special 
meaning.  We  give  an  example  that  demonstrates  how  one 
can  evaluate  and  derive  measures  to  be  most  useful.  Finally, 
we  Hit  the  many  methods  (e.g.,  cost  finding,  surveys,  etc.) 
that  can  be  used  for  obtaining  each  measure. 

Part  2  of  the  Manual  begins  with  Chapter  4.  This  part 
concerns  Evaluation  of  Information  Center  Operational 
Performance.  Operational  performance  can  involve  any 
level  of  the  hierarchy  of  functions,  services  and  products, 
aedvities,  resources  and  their  characteristics  as  shown  in 
Figure  3.  At  each  level,  as  mentioned  above,  there  is  a 
measurable  input  and  output.  Operational  performance 
includes  all  the  measures  and  derived  measures  associated 
with  input  and  output.  The  output  quantides  of  some  user 
related  services  and  products  (e.g.,  reference  requests, 
interlibrary  loans,  etc.)  arc  demand  driven.  That  is,  users 
determine  the  amounts  that  are  to  be  performed  or 
produced.  The  demand,  in  turn,  is  dependent  on  a  number 
of  factors  such  as  output  attnbutes  (e.g.,  price,  quality, 
timeliness,  etc.),  distance  to  the  center,  number  of  hours  of 
opening,  awareness  of  services,  competition  for  services  and 
so  on.  The  extent  to  which  these  factors  affect  demand  for 
services  is  the  focus  of  Part  Three.  Part  TVvo  deals  only  with 
operational  aspects  of  information  centers.  As  shown  in 
Figure  3,  all  levels  of  the  hierarchy  affect  performance. 
Examples  of  methods  for  evaluating  the  performance  of 
staff  and  automated  systems  are  given  in  Chapter  3. 
Measures  are  described  for  60  major  functions  performed 
in  information  centers  and  examples  of  staff  productivity  are 
given  for  each.  Finally,  information  center  funders  and 
managers  are  becoming  increasingly  concerned  with  quality 
assurance.  In  Chapter  6  we  provide  a  description  and 
examples  for  a  formal  Quality  Assurance  Surveillance 
Program  including  quality  control  of  staff-related  activities. 
Examples  are  also  given  on  how  to  do  quality  assurance. 

Both  operational  performance  and  effectiveness  evaluation 
often  involve  statistical  survey  or  sampling  methods.  For  this 
reason  we  have  devoted  an  entire  chapter  to  this  topic,  — 
Statistical  Survey  Methods  for  Evaluating  Information 
Center  Effectiveness  (Chapter  7).  For  surveys  we  provide 


detailed  descriptions  of  data  collection  methods,  sample 
design,  questionnaire  design,  data  processing,  data  analysis, 
statistical  analysis  and  presentation.  Strengths  and 
weaknesses  arc  presented  for  alternative  data  collection 
methods  such  as  self-administered  questionnaires, 
observation  of  users  in  an  information  center,  telephone 
interviews,  personal  interviews,  indepth  focus  interviews 
and  group  interviews.  Various  statistical  sample  designs  and 
sampling  methods  are  discussed  with  some  examples  given. 
Sampling  and  nonsampling  error  are  described  with  several 
numeric  examples  presented  in  detail  including  confidence 
intervals  and  sample  sizes  necessary  to  achieve  desired 
levels  of  statistical  precision.  Nonsampling  error  is  due  to 
communication  and  processing  failures.  These  errors  result 
from  poor  questionnaire  design,  inadequate  sample  frames, 
nonresponse  errors,  response  errors  and  other  sources  of 
error.  Various  means  lor  avoiding  these  errors  are  discussed. 

In  Chapter  8  —  Evaluating  the  Effectiveness  of  Information 
Centers;  Chapter  9  —  Evaluating  the  Effectiveness  of 
Specific  Information  Center  Services;  and  Chapter  10  — 
Evaluating  the  Higher  Order  Effects  of  Information  and 
Information  Service  Use,  we  give  detailed  sample  methods 
and  suggested  questions  that  can  be  used  on  a  survey 
questionnaire.  With  each  question  we  discuss  why  the 
question  might  be  asked,  problems  with  obtaining  the  data 
or  information  and  typical  responses  that  might  be  expected. 
The  data  are  used  to  determine  amount  of  use,  satisfaction 
with  services  and  attributes  of  services,  purpose  of  use, 
alternative  sources  of  services  and  potential  cost  of  using 
them  and  consequences  of  use.  Higher  order  effects  involve 
how  information  and  services  affect  users’  work  in  terms  of 
their  input  costs  and  output  quality,  timeliness  and  other 
consequences.  Three  example  services  are  dealt  with  in 
particular  detail:  online  bibliographic  and  numeric  database 
searching.  Current  Awareness  Bulletins  and  journal  routing. 

In  Part  4  —  Evaluating  Information  Center  Cost,  Benefit 
and  Value,  we  show  how  one  can  link  service  output 
attributes  to  user  satisfaction  and  extent  of  use  as  well  as  the 
higher  order  consequences  of  services.  We  also  deSenbe  and 
provide  an  example  of  a  particularly  powerful  method  and 
model  called  conjoint  measurement.  This  model  provides  a 
means ' .  determining  the  relative  utility  of  service  attributes 
such  as  price,  quality  and  timeliness  (Chapter  II  —  Relating 
Information  Center  Performance  to  Effectiveness).  A 
special  type  of  evaluation  involves  cost  and  benefit  (Chapter 
12).  We  consider  costs  and  benefits  to  be  unfavorable  (costs) 
and  favorable  (benefits)  comparisons  of  alternatives. 
Comparisons  would  be  done  for  input  costs,  output 
quantities  and  attributes,  effectiveness  measures  and  higher 
order  effects.  An  example  of  this  kind  of  cost  and  benefit 
analysis  is  given  for  a  resourc,.  (staff)  and  a  service  (online 
bibliographic  search). 

Finally,  we  devote  a  chapter  to  The  Value  of  Information 
Centers  and  Their  Services  (Chapter  13).  Examples  are 
given  for  three  perspectives  on  value.  The  first  perspective  is 
what  users  are  willing  to  pay  for  the  service;  particularly  in 
terms  of  their  ov.'n  time.  This  value  demonstrates  that  users 
consider  the  information  provided  by  information  centers  to 
be  of  considerable  value,  usually  three  to  eight  times  the  cost 
of  the  centers.  Then  we  look  at  how  much  more  it  would  cost 
users  to  obtain  the  information  (or  services)  if  there  were  no 
information  center.  This  value  is  also  usually  considerably 
higher  than  the  cost  of  the  center.  Finally,  we  establish 
measures  of  cost  avoidance  lost  without  services.  This  value 
is  usually  even  higher. 
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Wc  stress  that  the  evaluation  measures,  models  and  methods 
presented  m  this  Manual  are  not  the  only  ones  that 


evaluators  use.  The  reader  is  referred  to  other  approaches 
demonstrated  by  the  bibliography  at  the  end  of  the  Manual. 


Fig.3  Conceptual  framework  for  library  staffing  studies 


( 


10 


Chapter  2 

A  Framework  for  Evaluating  Information  Centers 


2.1  Background 

Several  studies  by  King  Research  ’  and  others  in  recent  years 
*■''  have  clearly  demonstrated  that  information  is  extensively 
read  and  used  by  professionals  such  as  scientists,  lawyers, 
engineers,  administrators,  and  so  on.  Readings  of  such 
information  arc  used  for  many  purposes,  including  research, 
writing,  proposal  developn.cnt,  management,  marketing, 
professional  development,  education,  and  so  on.  Each 
purpose  of  use  has  some  value,  otherwise  professionals 
would  not  devote  their  scarce  time  to  obtaining,  reading  and 
using  information.  One  contributor  notes  value  of 
information  is  in  cost  avoidance;  for  example,  by  not  having 
to  repeat  research  for  which  results  are  already  available  or 
by  avoiding  a  costly  environmental  control  penalty.  In  the 
literature  and  through  one's  own  experience,  anecdotal  data 
concerning  such  savings  are  available,  particularly 
concerning  very  large  savings.  Tlte  King  Research  studies 
mentioned  above  have  attempted  to  estimate  the  total  extent 
of  such  cost  asoidanco  by  observing  incidents  of  savings 
from  a  random  sample  of  scientists  and  determining  the 
consequences  of  their  readings  of  articles,  books  and 
technical  reports.  All  readings  in  1 984  of  articles,  books  and 
technical  reports  were  estimated  to  yield  S300  billion  in 
savings  to  the  scientific  community  in  the  US.'".  This  is  not 
surprising  when  one  considers  that  if  scientists  and 
engineers  were  denied  access  to  information,  it  is  doubtful 
that  they  could  accomplish  their  work.  Furthermore,  in 
addition  to  such  savings  attnbutable  to  information  and  its 
use,  we  have  observed  several  indicators  which  .show  that 
reading  also  contributes  to  improved  quality,  timeliness  and 
productivity  of  the  user’s  work. 

Another  way  to  emphasize  the  importance  of  leading  and 
keeping  up  with  current  research  findings  is  to  consider  the 
rapid  growth  of  science  and  technology.  The  amount  of 
recorded  literature  doubles  about  every  15  to  17  years.  This 
means  that  all  the  scientific  knowledge  recorded  throughout 
the  history  of  mankind  up  to  1970  has  now  more  than 
doubled  since  that  time  and  probably  will  double  again  by 
the  end  of  this  century.  This  means  that  engineers  or 
scientists,  upon  graduating  from  college,  will  be  exposed  to 
only  one-sixth  of  the  knowledge  that  they  must  master 
during  their  careers.  Because  of  the  necessity  to  keep  up 
with  the  literature,  scientists,  engineers  and  other 
professionals  read  a  large  number  of  articles,  technical 
reports  and  books.  Our  research  shows  that  all  professionals 
in  the  U.S.,  such  as  scientists,  engineers,  medical 
practitioners,  lawyers,  educators  and  businessmen,  read 
over  one  billion  journal  articles  a  year,  and  engineers  and 
scientists  account  for  nearly  one- third  of  that  amount'". 

Information  Centers  that  serve  companies,  government 
agencies,  academic  institutions,  other  organizations,  and 
individuals  have  been  undergoing  tremendous  change.  At 
one  time,  such  centers  were  concerned  largely  with  books, 
then  they  added  serials  and  periodicals.  Now  they  must  also 


store  and  provide  access  to  government  publications  and 
internal  research,  technical  and  other  reports  and,  because 
of  new  technology,  'hey  provide  access  to  audiovisual 
materials,  equipment  and  software. 

A  number  of  other  factors  are  changing ’the  needs  and 
requirements  of  information  centers  and  their  users.  They 
must  provide  access  to  an  accumulating  body  of  recorded 
knowledge.  As  mentioned  above,  since  1970  the  amount  of 
knowledge  recorded  throughout  the  history  of  mankind  up 
to  that  time  has  more  then  doubled.  One  of  the  unique  roles 
of  information  centers  and  documentation  centers  is  to 
provide  access  to  all  of  the  published  literature.  Naturally, 
they  cannot  each  hold  all  of  the  literature  ever  published, 
nor  even  all  published  in  a  particular  subject  area,  but  they 
do  provide  access  to  it  through  inlennstimtional  borrowing 
from  other  centers  or  thioiigh  document  delivery  services. 
Thus,  not  only  do  information  centers  have  to  cope  with  an 
ever  increasing  amount  of  new  literature,  they  currently  have 
to  provide  access  to  over  twice  as  large  a  body  of  hiei  aturc  in 
1990  as  they  did  in  1970. 

Information  Center  users  arc  becoming  more  information 
intensive  in  general  and  are  using  such  centers  more 
frequently  now  than  they  have  in  the  past.  For  example,  from 
1977  to  1984  we  observed  from  national  snivel's  that 
engineers  and  scientists  have  increased  the  proportion  of 
their  reading  from  information  center  copies  of  journals  by 
nearly  50  percent'".  Furthermore,  they  use  many 
information  services  now  that  were  not  even  known  25  years 
ago  such  as  online  retneval  of  bibliographic  and  numeric 
data.  They  also  provide  other  special  information  services 
such  as  translation  from  foreign  languages. 

Infoimation  Center  users  are  increasingly  exposed  to  new 
information  technology.  There  has  been  a  great  deal  of 
discussion,  in  the  literature,  of  how  as  the  usi  r  becomes 
more  “computer  and  information  literate,”  the  information 
center  will  no  longer  have  a  role  to  play  as  intermediary 
However,  the  opposite  is  observed  in  most  environments. 
Admittedly,  users  of  information  do  perform  some  of  their 
own  information  searches,  particularly  when  they  have 
become  comfortable  with  available  systems.  However,  ,ts 
they  do  more  searching,  they  recognize  that  more  of  their 
scarce  time  is  being  taken  up  with  information  searching  and 
retrieval  activities.  They  learn  that  there  are  a  large  number 
and  variety  of  sources  of  information  to  choose  from,  and 
that  the  sources  change  over  time  in  terms  of  coverage, 
procedures  for  use,  quality,  etc.  Once  they  recognize  the 
complexity  of  information  searching  and  retrieval,  thfy 
begin  to  return  to  the  information  center  as  an  intermediary, 
particularly  for  their  more  complex  requirements.  However, 
since  their  experience  has  given  them  a  better  awareness  and 
understanding  of  information  systems,  they  are  better  able 
to  articulate  their  information  needs,  and  they  are  more 
sophisticated  in  terms  of  their  expectations  of  and  demands 
on  mformation  center  services. 
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Because  of  the  evolution  of  the  inforniation-seeking 
behavior  of  professionals,  and  the  recognition  of  the 
importance  and  value  of  information  and  information 
services,  many  organizations  arc  evaluating  their 
information  services  to  ensure  that  sufficient  services  are 
being  provided  and  that  optimum  return-on  investment  in 
these  services  is  being  achieved.  This  chapter  presents  an 
evaluation  framework  which  can  be  used  to:  (1)  assess  the 
information-seeking  behavior  of  professionals,  (2)  establish 
indicators  of  extent  to  wh<ch  use  of  information  affects  their 
work,  (3)  evaluate  the  perfonnance  of  information  center 
operations  and  services  and  (4)  determine  the  extent  to 
which  information  center  services  contribute  to  the  use, 
usefulness  and  value  of  information.  By  determining  the 
extent  to  which  services  contribute  to  users’  work, 
information  managers  can  perform  a  truly  comprehensive 
evaluation  of  operations  and  services.  Such  comprehensive 
evaluation  establishes  not  only  measures  of  input  costs  and 
output  quantities  and  attributes  of  all  operational  functions 
and  -erviccs,  but  also  the  relationships  between  (1)  input 
cost  and  output  and  (2)  between  output  attributes  and  the 
extent  to  which  services  are  used  and  the  effects  of  the 
services  on  users’  work.  We  have  found  that  information 
center  funders  and  high  level  managers  relate  very  well  to 
this  way  of  evaluating  operations  and  services. 

2.2.  Framework  for  a  Comprehensive  Evaluation  of 
Information  Centers  and  Services 

Below  we  present  an  approach  to  data  collection  and 
analysis  for  information  center  evaluation  planning  and 
management  through  an  example.  The  framework  for  this 
example  is  displayed  in  Figure  4.  There  are  three  dimensions 
to  the  framework.  One  dimension  involves  three 
perspectives  on  evaluation;  the  information  center  service 
perspective  (e.g.,  online  searching),  user  perspective  (c.g.,  a 
scientist  conducting  research),  and  the  user’s  organization 
perspective.  Decisions  made  based  on  any  one  of  the  three 
levels  will  effect  decisions  based  on  the  other  two 
perspectives.  For  example,  if  information  center 
management  decides  to  improve  its  online  searching  by 
hiring  more  competent  information  specialists,  the  quality 
and  timeliness  of  the  searches  should  improve.  Evidence 
shows  that  improved  service  should  result  in  (1)  more 
searches  being  performed,  (2)  users’  time  being  saved,  and 
(3)  improvements  in  output  of  users’  work.  Since  users 
contribute  to  the  organization’s  goals  and  mission,  any 
change  in  their  work  wiil  have  some  effect  on  the  goals  and 
mi.ssion.  On  the  other  hand,  if  users  change  the  nature  of 
their  work  or  alter  their  information-seeking  behavior,  their 
infornation  needs  and  requirements  will  change  thereby 
affecting  the  information  center’s  ability  to  serve  them. 

We  look  at  all  three  perspectives  as  in  solving  functions  {or 
services)  which  consist  of  activities  necessary  to  perform  the 
functions  and  resources  that  are  neces.5ary  to  perform  the 
activities  (see  Chapter  3  for  a  more  detailed  description  of 
these  levels).  This  is  the  second  dimension  of  the  evaluation, 
framework  in  Figure  4.  From  the  center’s  perspective,  online 
searching  is  a  service;  activities  might  include  interviewing 
users,  negotiating  the  search,  developing  a  starch  strategy, 
conducting  a  search,  reviewing  results,  providing  results  to 
users,  etc.  Resources  could  include  staff,  terminals, 
photocopiers,  vendor  services,  reference  materials,  etc. 
From  the  user’s  perspective,  functions  might  include 
research  engineering,  management,  legal  work, 
professional  development,  etc.  Resources  would  include  the 


users’  time,  information,  support  staff,  equipment,  etc.  used 
to  perform  these  functions.  From  the  organization’s 
perspective,  functions  might  be  R&D,  manufacturing, 
marketing,  etc.  and  resources  are  the  sum  of  staff,  facilities, 
equipment,  etc.  applied  to  the  functions. 

Each  function  or  service,  activity  and  resource  has  an  input 
cost  and  an  output  associated  with  it.  The  inputs  and  outputs 
are  the  third  dimension  of  the  evaluation  framework.  We 
refer  to  the  application  of  these  resources  (or  resource 
funds)  as  input  costs.  The  output  of  the  ftinctioiis  (or 
services),  activities  or  application  of  resources  is  some 
product  or  service.  For  example,  the  output  of  online 
searching  can  be  printouts  or  search  results  communicated 
orally  by  the  searcher.  The  output  of  users’  work  may  be 
documentation  of  their  work  in  the  form  of  laboratory  notes, 
technical  reports,  proposals,  presentation,,  etc.  Thus, 
information  serves  as  both  input  and  output  for  users’  work 
as  discussed  in  more  depth  in  Chapter  4.  In  addition  to 
identifiable  output  products  or  services,  the  outputs  can  also 
be  characterized  by  attributes  such  as  quality,  timeliness,  etc. 
of  the  services  or  work  performed. 

For  the  information  center  example  in  the  Figure  4,  the 
.service  is  online  searching  and  resources  aie  searchers, 
terminals,  search  tools,  support  staff  (e.g..  to  do 
photocopying,  typing,  etc.),  communication,  search  systems, 
photocopying,  etc.  Associated  with  the  resources  are  cost.- 
whieh  arc  dependent  on  the  attributes  of  the  resources 
(arrow  |a)).  Output  quantities  are  related  to  costs.  Likcwi.s.;, 
output  (in  terms  of  number  of  searches  conducted,  quality 
and  timclines.s)  depends  partially  on  the  attributes  oi  the 
resources  (arrow  |b)).  The  relationships  between  resource 
attributes  and  input  cost  and  output  performance  are  quite 
clear.  For  example,  better  searcher  competencies  (i.e., 
knowledge,  skills  and  attitudes)  usually  cost  more  (in 
salaries)  but  should  also  yield  more  and/or  better  search 
output.  Performance  of  online  searching  can  also  be 
measured  by  cost  per  search  or  productivity  (searches  per 
hour  or  dollar  cost).  These  relationships  are  also  designated 
by  arrow  (c).  In  considering  productivity,  we  feel  that  it  i.s 
best  to  incorporate  quality  and  timclii’.e'.s  as  well  as 
quantities  produced,  for  example,  by  measuring  cost  per 
search  at  discrete  levels  of  quality  and  timeline.s;  or  at  least 
above  an  acceptable  level  of  quality  and  timeliness. 

Similarly,  laboratory  research  conducted  by  a  scientist  is 
given  as  an  example  of  an  activity  from  the  user  perspective. 
To  perform  this  research  activity,  scientists  or  other 
professionals  must  have  such  resources  as  their  own  time, 
equipment,  instrumentation,  facilities,  support  staff  and 
information.  The  amount  and  quality  of  information  used  by 
the  scientists  depends  partially  on  the  output  attributes  of 
the  online  search.  This  relationship  is  designated  by  arrow 
(f).  In  turn,  the  amount,  quality  and  timeliness  of 
information  will  affect  cost  of  performing  the  research 
activity  (arrow  |g|)  and  output  amount,  quality  and 
timeliness  of  research  (arrow  (hj).  Productivity  of  the 
scientist  is  shown  by  (arrow  (i)).  'Thus,  the  output  of  the 
information  service  should  affect  the  productivity  of 
scientists.  Similarly  the  outputs  of  scientists'  activities 
should  .  >(cct  the  input  costs  and  outputs  of  their  units,  and 
hence,  the  users’  organization  as  a  whole  (arrows  (k),  (1 1,  and 
|m]).  Furthermore,  the  organization’s  total  input  cost  is 
affected  by  the  center’s  cost  (arrow  |d))  and  user's  cost 
(arrow  (jj). 


SHE7/XCE  OR 

EERSPECriVE  INPUT  ACTr.’IW/RESOURCES  OUTPUT 


(A) 

CENTRE  Online  Searching/ 

SERVICES  SeaiECher,  Termincil , 

HTOtoccpy,  etc. 


Cost  Per:  No.  of  searches, 

(d)  Hours  of  searcher,  (c)  Quality,  (e) 


Fi9.4  Framework  for  evaluating 
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Measures 

For  each  information  service  one  can  define  siibactivities 
and  resources  that  are  used  in  providing  the  services  (A  in 
Figure  4).  In  addition  to  the  numbers  or  amounts  or  the 
resources,  one  can  characterize  them  in  teims  of  attributes 
that  are  likely  to  affect  input  cost  (B)  and  output  quantities 
and  attributes  (C).  Such  attributes  include  staff 
competencies  (knowledge,  skills,  attitudes)  or  indicators  of 
competencies  such  as  educational  background,  training  and 
years  of  experience;  equipment  capabilities  such  as 
communication  rates,  storage  capacities,  reliability,  etc.; 
photocopy  or  reproduction  quality;  microfilm  viewer 
readability;  and  so  on.  Cost  data  can  be  subdivided  into 
fixed  onetime,  fixed  recurring  and  variable  elements  and  by 
direct  and  indirect  costs  (see  Chapter  4).  Output  (C)  can  be 
identified  and  measured  in  terms  of  quantities  produced, 
quality  (probably  more  than  one  measure  per  .service), 
timeliness  (probably  both  by  response  times  and  by  the 
difference  between  negotiated  time  and  actual  time  the 
service  is  delivered)  and  any  other  attributes  that  might 
appreciably  affect  user  satisfaction  and  continued  use  of  the 
service.  We  have  found  that  it  is  sometimes  useful  to  set  an 
acceptable  level  of  output  quality  and  timeliness  so  that  one 
can  estimate  the  frequency  of  unacceptable  performance 
and,  perhaps,  later  implement  a  quality  control  program  for 
some  services  (see  Chanter  6) 

In  particular,  we  suggest  that  evaluators  should  identify  <hc 
functions  (or  activities)  performed  by  users  and  determine 
the  extent  to  which  they  arc  performed  (i.e.,  in  terms  of 
hours  spent)  by  users  (D).  For  each  function,  the  resources 
used  are  determined,  particularly  in  terms  of  the  types  of 
information  provided  by  the  information  center.  One  can 
also  collect  data  on  attitudes  regarding  the  relative 
importance  of  information  compared  with  other  resources 
used  in  performing  the  various  user  functions.  The  cost  (E) 
to  users  of  identifying,  acquiring  and  reading  information 
can  be  measured.  These  costs  include  user  labor,  support 
labor,  equipment,  purchases  of  materials,  information,  and 
so  on.  One  can  also  measure  indicators  of  user  output  (F) 
associated  with  each  function.  Such  output  includes  number 
of  proposals  or  plans  written,  research  reports  (c.g.. 
laboratory  notebooks)  prepared,  journal  articles  or  other 
publications  written,  consultations  performed,  and  so  on.  In 
addition,  for  critical  incidents  one  can  estimate  the 
contnbution  that  readings  (of  books,  journal  articles, 
technical  reports,  etc.)  make  to  savings  in  labor  and 
equipment,  improving  quality  or  speeding  up  completion  of 
an  activity,  and  so  on.  Finally,  to  the  degree  possible,  one  can 
measure  how  user  performance,  in  turn,  affects  the  unit's  or 
the  organization's  total  input  costs  and  output. 

Models 

One  can  establish  correlations  and  other  quantifiable 
relationships  between  information  center  input  costs  and 
outputs;  service  outputs  and  user  input  costs  or  outputs; 
user  input  costs  and  user  outputs;  and  user  outputs  and  unit/ 
organization  input  costs.  These  relationships  or  links  can  be 
developed  largely  through  correlation,  multiple  regression 
and  conjoint  measurement  models.  Those  models  can  be 
applied  using  surveys  (see  Part  3)  because  the  surveys  yield 
several  hundred  observations  of  critical  incidents.  Conjoint 
measurement  permits  one  to  determine  the  relative 
contributions  that  service  performance  attributes  make  over 
various  levels'  of  the  attributes.  For  example,  one  can 
determine  how  much  use  would  be  lost  over  various 


combinations  of  search  quality  and  speed  of  response  (sec 
Chapter  H). 

Other  independent  variables  are  introduced  to  account  for 
other  sources  of  variation.  Our  experience  has  shown  that 
the  measurable  relationships  become  weaker  as  the  ability 
to  observe  or  measure  becomes  more  difficult.  Nevertheless, 
we  have  found  positive  correlations  in  most  such  instances. 
Cost  and  benefit  analysis  is  performed  by  comparing  each 
information  service  to  its  least  expensive  alternative  (or 
alternative  that  is  most  likely  to  be  employed  by  users), 
nicn,  by  application  of  the  im  dels,  one  can  compare  the 
current  sendee  with  the  alternative  in  terms  of:  service  input 
costs,  service  outputs,  user  input  costs,  user  outputs,  and  so 
on. 


2.3.  Definition  of  Terms  Used  in  the  Manual 


In  recognition  that  evaluation  terms  have  different 
definitions  and  meanings  in  various  professional  fields  or 
specialties,  we  offer  the  following  definition  of  terms.  These 
definitions  arc  provided  to  ensure  a  common  basis  for 
reading  this  manual. 

•  Information  Center—  An  information  center  is  defined 
as  a  unit  whose  principal  function  is  to  provide 
information  services  on  products  for  the  benefit  of 
cndiisers  and/or  intermediary  organizations. 
Information  centers  include  organization  libraries  (i.c.. 
special  libraries,  academic  libraries,  etc.), 
clearinghouses,  technical  irformation  centers,  special 
publication  units,  and  so  on. 

•  Operational  Function  or  Senicc  —  Information  center 
operational  functions  and  services  may  be  defined  at 
several  levels.  Examples  of  types  of  functions  and 
services  include  (1)  technical  functions  such  as 
ordering  or  cataloging  information  materials,  etc.,  (2) 
user-related  services  such  as  reference  or  access  to  a 
collection,  or  (3)  support  function  provided  by 
administrators.  Operational  functions  and  services  can 
themselves  be  subdivided  into  the  discrete  activities 
necessary  to  perform  them  (e.g.,  contacting  a  patron, 
negotiating  what  is  needed,  performing  online 
searches,  reporting  results,  etc.).  Each  activity  may 
require  several  resources  (e.g.,  staff,  equipment,  etc.). 
Products  are  usually  the  physical  output  of  services 
(e.g.  pnnt-out  of  an  online  search),  althougli  every 
service  docs  not  necessarily  have  a  product. 

•  Input — Several  input  resources  arc  necessary  to  offer/ 
jjcrform  services.  These  include  capital,  staff, 
equipment,  facilities,  information,  supplies, 
administrative  and  support  staff,  etc.  Input  is 
considered  the  application  of  these  resources.  Each 
resource  can  be  measured  in  several  ways;  for  example, 
staff  can  be  measured  in  Full-Time  Equivalents, 
number  of  persons,  hours  of  work,  or  cost  (salary, 
fringe  benefits,  overhead). 

•  Output  —  Measures  of  output  of  information  services 
include  quantities  of  output  (e.g.,  number  of  items 
acquired,  number  of  items  circulated,  number  of 
searches  performed,  etc.)  and  output  attributes  such  as 
quality  of  service,  timeliness  of  services  provided, 
availability  and  accessibility  of  informatior.  center 
materials,  equipment,  staff  and  facilities,  etc. 

•  Productivity  —  Productivity  is  a  measure  of  the  ratio  of 
output  divided  by  input.  It  is  formally  defined  as  “a 
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concept  that  expresses  the  relationship  between  the 
quantity  of  goods  and  services  produced— output— 
and  the  quantity  of  labor,  capital,  land,  energy,  and 
other  resources  that  produced  it— input.”  (US. 
Department  of  Labor,  Bureau  of  Labor  Statistics, 
Productivity  and  the  Economy:  A  Chartbook  (Bulletin 
2172).  Washington,  DC:  Government  Printing  Office, 
June  1983.)  Productivity  is  a  derived  measure  which 
links  input  and  output.  'Die  weakness  of  the  traditional 
measure  of  produc.ivity  is  that  there  is  an 
interdependence  among  amount  of  input  resources, 
input  resource  attributes,  output  quantities  produced, 
and  output  attributes  such  as  quality,  timeliness,  etc. 
For  example,  a  center  staff's  competency  will  affect 
both  input  (amount  of  time  and  cost  necessary  to 
perform  a  service)  us  well  as  output  (amount,  quality 
and  timeliness  of  service).  There  is  usually  an  inverse 
relationship  between  input  amount  of  services 
produced  and  scr*  ice  output  quality,  timeliness,  etc. 


•  Performance  —  Performance  is  an  indication  of  how 
well  a  service  or  activity  is  performed.  It  can  be 
measured  in  terms  of  the  input  costs  and  output 
quantities  produced,  quaiity,  timeliness,  availability, 
accessibility,  etc.  Other  indications  of  performance  are 
productivity,  efficiency  (ic.,  how  close  services  or 
activities  cjinc  to  aciiicving  some  maximum),  etc. 
Output  atiribules  relates  to  the  cffcc'iveness  of 
services  (see  below).  Mea.sures  of  productivity, 
efficiency,  etc.,  are  measures  which  are  internal  to  the 
service  and  they  can  be  used  to  help  manage  an 
information  center. 

•  Effectiveness  —  Effectiveness  is  measured  from  the 
perspective  of  the  users  of  information  center  services. 
Examples  measures  of  effectiveness  include  user 
satisfaction  wih  a  service,  repeated  use  oia  service  and 
number  of  times  a  service  is  used  (first  order  effects). 
Higher  order  effects  include  effect  of  services  on  the 
user's  research  and  consequences  of  the  user's 
research,  improved  pre  activity  of  the  user's  operation 
or  research,  etc.  Presumably,  improved  service  quality, 
timeliness,  availability,  etc.  should  result  in  greater  user 
satisfaction,  repeated  use  and  the  number  of  times  a 
service  is  used.  Thus,  information  center  service 
output  attributes  affects  the  effectiveness  of  the 
service. 


•  Cost  and  Benefits  —  The  costs  (i.e.,  detriments)  and 
beneffts  of  an  information  center  service  are  the 
unfavorable  and  favorable  comparisons  of  a  service 
with  alternative  services,  in  terms  of  differences  in 
input,  output,  pcnurmancc  and  effectiveness.  For 
example,  the  value  added  by  a  service  could  be  that  the 
service  costs  less  than  an  alternative  service,  and/or  the 
service  is  better  than  the  alternative  in  terms  of 
performance  or  effectiveness.  Such  an  alternative  to  a 
service  might  be  for  users  to  perform  the  work 
themselves  or  to  engage  a  consultant  or  company  to 
provide  the  service  to  them. 

•  Indicators  —  Sometimes  it  is  not  possible  to  measure 
input,  output,  performance  oi  effectiveness  directly. 
Thus,  indicators  must  suffice.  Indicators  of  a  center 
staff  member's  competency  are  degrees  held, 
university  attended,  professional  awards  given,  elected 
position  in  a  professional  society,  etc.  Indicators  may 
be  needed  for  higher  order  effects.  For  example,  one 
indicator  of  research  output  is  number  of  laboratory 
notes  or  articles  written  which  report  the  research. 

•  luctors  That  Affect  Information  Center  Service  Input, 
Output,  Performance  and  Effectiveness  —  Factors  that 
affect  input  might  include  staff  characteristics, 
equipment  attributes,  etc.  Examples  of  factors  that 
mignt  affect  output  or  performance  (in  addition  to 
amount  of  input  resources)  are  management  (policies, 
capabilities,  attitudes,  ete.),  physical  environment, 
attitude  or  capability  of  users,  etc.  Factors  that  might 
affect  effectiveness  (m  addition  to  output 
performance)  are  user  awareness  of,  attitudes  toward, 
or  perception  of  an  information  center,  a  charge  for  the 
service,  distance  to  the  service,  communication 
constraints,  etc.  Some  factors  arc  internal  and, 
therefore,  controllable.  Other  factors  arc  external  and 
less  controllable  by  Information  center  management. 

•  Linkage  of  Information  Center  Services'  Input  to 
Output,  and  Center  Service  Performance  to 
Effectiveness  —  Linkage  is  achieved  through 
correlation  and/or  mathematical  models  which  show 
that  (I)  output  quantities  and  attributes  are  related  to 
input  resources  in  addition  to  other  factors,  (2)  first 
order  effects  (e.g,,  frequency  of  use  of  a  service) 
depend  on  output  attributes  in  addition  to  other 
factors,  and  (3)  higher  order  effects  depend  on  first 
order  effects. 
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Chapter  3 

Evaluation  Measures,  Models  and  Methods 


3.1  Background 

The  heart  of  the  evaluation  of  information  centers  involves 
measures,  models  and  methods.  In  this  chapter  (partially 
adapted  from  Keys  to  Success)  we  discuss  some  concepts  of 
these  three  components  of  information  center  evaluation. 
As  stated  earlier,  the  concepts  we  present  are  by  no  means 
the  only  wtty  of  looking  at  evaluation  nor  are  the  measures, 
models,  and  methods  the  only  ones  used  for  evaluating 
information  centers  and  services.  However,  over  the  years 
the  approach  presented  here  has  been  found  to  be  useful  for 
evaluating  a  large  number  (about  300)  of  information 
centers,  libraries,  clearinghouses,  publications,  online 
services,  centralized  and  decentralized  automated  systems, 
library  networks,  and  a  host  of  other  information  services. 

Evaluation  mcasui  -s  are  the  “values”  measured  or  placed 
on  information  cemeis  operations  and  services.  In  Chapter 
2  we  presented  a  particular  framework  for  evaluation.  Here 
we  present  examples  of  measures  related  to  (1)  input  cost 
(i.c.,  amounts  oi  lesourees  or  money  applied  to  operational 
functions  or  services  and  attributes  of  these  resources),  (2) 
output  (i.e.,  the  quantities  of  operational  functions  or 
services  produced  and  attributes  of  the  output),  (3) 
effectiveness  (i.e.,  satisfaction  with  services,  amount  of  use, 
purpose  of  use,  consequences  of  use),  and  (4)  domain  (i.e., 
descriptions  of  the  environment  served  by  the  information 
center). 

Tile  individual  measures  alone  do  not  convey  much 
meaning.  For  example,  knowing  that  staff  input  costs  are 
$140,000  for  online  bibliographic  search  services  is  not 
particularly  useful  without  some  context  such  as  timeframe, 
number  and  experience  of  staff,  number  and  type  of 
searches  performed,  etc.  For  this  reason,  we  present  some 
relationships  among  measures  which  we  refer  to  as  models. 
These  models  provide  more  useful  tools  for  evaluation  and 
decisionmaking  than  measures  alone.  The  simplest  of 
relationships  are  ratios  (e.g.,  number  of  items  produced  per 
hour  of  labor)  or  averages  (e.g.,  average  number  of  searches 
conducted  per  user).  More  complex  models  include 
statistical  correlation,  multiple  regression  and  conjoint 
measurement,  and  operations  research  models  including 
queuing,  Markoff  chains,  etc.  As  Meadows'  points  out,  there 
are  several  other  types  of  models  as  well,  including  non- 
quantitative  models.  The  reader  is  referred  to  his  paper  and 
those  of  others  to  become  familiar  with  the  range  of  models 
available  for  evaluation. 

In  this  Manual  we  focus  on  three  particular  methods:  (1) 
cost  finding  for  measuring  input  costs  and  output,  (2)  quality 
assurance  surveillance,  and  (3)  statistical  surveys.  Cost 
finding  involves  nonexact,  but  very  useful  methods  for 
measiiting  costs  and  output  quantities.  A  chaptei  is  devoted 
to  measuring  output  quantities  and  attributes  such  as  quality 
and  timeliness  in  order  to  assure  that  acceptable  levels  are 
continually  achieved.  We  also  devote  several  chapters  to 
statistical  surveys  since  they  are  used  to  measure  the 
effectiveness  of  information  centers  and  services.  Finally,  in 


Part  4,  we  discuss  cost  and  benefit  analysis  and  measures  of 
the  value  of  information  centers  and  services.  The 
remainder  of  this  chapter  discusses  some  of  the  concepts  of 
evaluation  measures,  models  and  methods. 

3.2.  Evaluation  Measures 

Definition  of  Measure 

For  the  purpose  of  this  manual  we  define  “measure"  as 
follows: 

Measure:  Genetically  used  to  mean  any  process  for 

describing  in  quantitative  values;  things, 
people,  events,  etc.  Measure  also  means  the 
value  being  measured. 

In  the  Manual  we  use  measures  as  a  generic  term  which 
reflects  a  number  of  commonly  used  measures  in  evaluation 
such  as: 

To  measure  distance  (e.g.,  the  number  of  feet  or  meters  of 
shelving)  or  the  square  feet  (meters)  of  floor  space. 

To  count  the  number  of  people  on  the  staff  or  number  of 
visits  to  an  information  center. 

To  record  the  duration  of  the  time  required  to  respond  to 
online  search  requests  or  the  amount  of  time  required  to 
actually  perform  a  search. 

To  observe  what  users  or  staff  are  doing  at  a  point  in  time 
such  as  doing  research  at  study  tables  or  reshelving 
documents. 

To  suney  users  to  determine  their  satisfaction  with  services 
or  their  number  of  uses  of  a  service  in  the  last  month. 

To  compute  costs  of  resources  or  services. 

In  this  manual  we  most  often  use  the  term  “measure”  to 
mean  the  values  being  measured  by  the  processes  of 
measunng  above.  There  are  two  components  of  measures: 

Numeric  Values  measured  such  as  12  staff  members,  12 
dollars,  12 hours,  12  searches  performed. 

Units  of  measure  such  as  12  staff  members,  12  dollars,  12 
hours,  12  searches  performed. 

Part  of  the  definition  of  evaluation  measures  includes  the 
context  within  which  the  measures  are  taken.  Such  a  context 
should  include  at  a  minimum: 

Method  used.  For  example,  if  a  survey  is  used  it  should  be 
described  in  some  detail  somewhere  in  the  report  and 
mentioned  in  any  tables  presenting  the  data  so  that  one  can 
refer  to  the  description. 
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Context  of  units  measured.  For  example,  if  staff  counts  arc 
given  it  is  important  to  specify  what  types  of  staff  (eg., 
professionals,  paraprofessionals,  clerical,  etc.,)  whether  the 
count  is  of  all  staff  (i.e.,  a  head  count),  full-time  equivalents, 
etc. 

Time  period  in  which  units  are  measured.  For  example,  if  cost 
finding  is  used  to  estimate  staff  cost  of  searches,  the  time  in 
which  cost  allocations  are  made  should  be  specified  (c.g.,  fall 
of  1990  or  March  1990). 

All  of  the  above  should  be  included  in  describing  or 
reporting  measures. 

Generic  Types  of  Measures 

Evaluation  of  information  centers  involves  four  generic 
types  of  measures  including  input  cost  measures,  output 
measures,  effectiveness  measures,  and  domain  measures. 
The  first  two  types  of  measures  (input  cost  and  output) 
involve  information  center  operations  and  they,  individually 
or  together,  help  to  establish  the  performance  of  resources, 
activities,  services,  functions  or  the  entire  information 
center.  Such  measures  include  the  amount  and  attributes  of 
resources  applied  to  services  and  output  quantity  and 
attributes  of  services.  Effectiveness  measures  are  those 
involving  the  effects  of  center  services  from  the  perspective 
of  users  such  as  amount  of  use,  purpose  of  use, 
consequences  of  use,  etc.  Domain  measures  involve 
descriptions  of  the  environme.'t  or  context  of  the 
information  center.  Such  measures  it.  elude  the  total  number 
of  persons  in  the  service  populatu  n,  their  information 
needs,  etc. 

Information  center  performance  is  largely  controlled  by 
center  management.  For  example,  a  management  decision 
to  hire  online  bibliographic  searchers  with  subject 
knowledge  affects  input  cost  (because  their  salaries  are  high) 
as  well  as  output  quantities,  quality,  timeliness,  etc.  If  mput 
costs  or  output  are  not  satisfactory,  management  can  correct 
the  situation  through  training  or  firing  and  re-hiring,  etc. 
However,  management  has  less  direct  control  over  center 
effectiveness  because  there  are  so  many  external  factors  that 
affect  the  extent  to  which  services  are  used  and  the 
consequences  of  that  use.  However,  management  can  react 
to  these  factors  with  proper  knowledge  of  them  and  how 
they  affect  use.  Managers  have  almost  no  control  over 
domain  measures.  Nevertheless,  knowledge  of  them  is  very 
important  to  center  managers  (and  staff  as  well).  Examples 
of  the  four  types  of  measures  are  given  below. 

Input  Cost  Measures 
Input  cost  measures  include; 

•  Amount  of  resources  applied  to  operational  functions 
and  services. 

•  Relevant  attributes  of  resources  appiied  to  services. 

•  Amount  of  money  appiied  to  sendees:  '.vheic  money 
(i.e.,  dollars,  pounds,  francs,  etc.)  is  a  common  unit  that 
can  be  applied  to  all  resources. 

Examples  of  quantities  and  attributes  of  resources  are 
as  follows; 


RfSOURCeS 

QUANTTnES 

AHreeUTES 

sun 

No  Of  poopio  (hoad 
coont),  no  of  hours 
W0f1(sd  Ail  uno  »qui 
vsisnis  (PTE's) 

(.tv*i  (1  • ,  profssstonsi 
pars  profs sslonaf. 
dSficaV^upport), 
compstsnes  (knowlsdgs 
skins,  sttiiudts}.  sducstion, 
yssfs  of  sxpsrisncs 

CC««CtiOA 

No  of  titlos  in 
cdsction,  no  of 
tuss  pufchassd, 
no  of  rtsms/piocos 
purchasod 

Typs  of  matsrlal  0 1 ,  books, 
psrtodici^,  sudKivisusis 
tschntcai  rsports).  Sk^jset 
dsssi*cstion  ms^s  (i  s . 
print/pspsr.  microfonn. 
sisctronic).  sgs 

Equipment 

No  of  pf«o«t  ol 
•quipmoni 

Typs  of  squipmsnt, 
spphOaiton.  sgs,  rsksbiiity. 
spssd  msks  sr>d  rnodsi 

FtcMits 

Afss  (floor  spset), 
no  of  Boors,  no 
of  SitSS 

Punctionsf  srsss,  mssi  vs 
sstsUtfs  branch,  and  spsdsJ 
attnbutss  such  ss  hsr>dicsp 
sntrsnes,  smbisnes,  ttc 

fifWidMi/Cctt 

Oossrs,  pounds, 
fftflCt,  stc 

Aifocstjons  of  fuTids,  appii- 
cst<on  of  funds,  dirset  vs 
ir>dirsct  nxsd  vs  vsrisbis  (sss 
Cnsptsr  4) 

Input  costs  of  operational  functions  and  services  are  the  sum 

of  all  resources  applied  to  the  functions  or  services. 

Output  Measures 

Corresponding  to  mput  costs  are  output  measures  which 

include: 

•  Quantities  of  output  of  services  and  operational 
functions  are  measures  of  the  numbers  of  transactions 
or  Items  provided.  Such  quantities  of  services  can 
include,  for  example,  number  of  searches  performed, 
number  of  documents  provided  or  made  available,  or 
number  of  documents  photocopied,  etc.  Examples  of 
operational  functioti  outputs  include  number  of  items 
catalogued,  number  of  items  ordered,  and  so  on. 
Quantities  of  output  should  correspond  to  the  same 
time  period  and  attnbutes  of  resources  as  were  used  to 
measure  the  service  input  costs.  Each  unit  of  output 
has  inherent  attributes  associated  with  it  such  as 
quality,  timeliness,  availability  and  accessibility  that 
should  be  measured. 

•  Quality  is  a  generic  output  measure  which  describes 
the  grade  or  “goodness”  of  information  center  services. 
Qu^ity  is  measured  less  frequently  for  services  than 
timeliness,  availability  and  accessibility.  Furthermore, 
it  is  difficult  to  identify  and  measure  quality  attributes 
for  many  services.  Examples  of  quality  measures 
include  relevance  of  search  outputs,  accuracy  of 
cataloging,  or  level  of  excellence  of  activities.  These 
measures  must  be  carefully  defined.  Sometimes  quality 
cannot  be  measured  directly,  in  which  case,  scale 
values,  (e.g.,  a  scale  of  1  to  5)  are  used.  Quality  should 
be  measured  for  specific  units  of  output,  (e.g.,  an  item 
cataloged,  a  reference  search,  etc.).  However,  all 
transactions  or  units  of  output  need  not  always  be 
measured  since  sampling  methods  can  be  used. 

•  Timeliness  of  user  services  involves  an  elapsed  time 
between  request  and  receipt  of  the  output  of  a  service. 
The  elapsed  time  can  be  measured  in  minutes  (e.g., 
with  circulations)  hours  or  days.  For  some  services 
(e.g.,  online  database  searching,  interlibrary 
borrowing,  etc.),  it  is  useful  to  establish  a  time  by  which 
users  require  a  response  and  measure  tlie  difference 
between  the  elaps^  time  and  the  required  time.  For 
example,  a  user  may  require  search  results  in  three  days 
to  prepare  a  report.  If  the  elapsed  response  time  is  four 
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daj's  the  difference  is  one  day  late.  Anoth.  aser  may 
require  the  results  in  five  days.  If  the  elapsed  response 
time  is  four  days,  this  user  will  be  much  happier  than 
the  user  who  requited  the  re.sponse  in  three  days.  Just 
as  with  quality,  timeliness  should  be  measured  for 
specific  units  of  output. 

•  A  principal  measure  of  availability  is  number  of  hours 
of  service,  (e.g.,  number  of  hours  of  ser^ce  per  day  or 
per  week),  or  number  of  person  hours  of  service,  (e.g., 
number  of  hours  of  service  times  the  number  of 
persons  providing  the  service  during  those  hours).  The 
spread  of  hours  of  availability  over  a  week,  (e.g., 
morning,  afternoon,  evening  and  weekend  hours)  is 
also  a  measure  of  availability.  Another  example  of 
availability  is  the  specified  loan  period  of  materials. 
Availability  is  usually  measured  for  a  service,  but  not  as 
a  unit  of  service  output. 

•  One  measure  of  accessibility  is  the  distance  of  the 
service  (or  information  center)  to  the  user.  Distance 
can  be  measured  in  feet  or  floors  in  a  building,  or  in 
terms  of  a  surrogate  measure  such  as  time  (e.g., 
minutes).  Waiting  time  (e.g.,  in  service  queues)  is  an 
important  measure  of  service  accessibility.  The  time 
taken  by  users  to  get  to  an  information  center  and 
waiting  for  services  is  a  portion  of  the  “price”  paid  by 
users  to  use  center  services.  The  more  users  are 
required  to  “pay"  in  terms  of  their  own  time,  the  less 
likely  they  are  to  use  the  services.  Accessibility  to 
collection,  equipment,  etc.  is  of  particular  concern  to 
users.  Materials  may  be  Inaccessible  if  kept  in  a  remote 
storage  or  compact  storage.  Public  access  terminals 
may  be  inaccessible  as  a  result  of  extensive  use.  Finally, 
accessibility  is  an  important  consideration  for  people 
with  disabilities.  Physical  accessibility  can  be  assessed 
in  terms  of  the  existence  of  special  facilities  such  as 
wheelchair  ramps  or  by  rating  the  degree  of 
accessibility  using  scales  (1  to  5).  Psychological 
barriers  to  using  an  information  center  can  also  create 
accessibility  problems.  Perceptions  of  accessibility  to 
the  center  and  its  services  on  the  part  of  the  population 
served  will  have  an  effect  on  the  amount  of  use  that  is 
made  of  them.  Distance  is  observed  for  the  entire 
center  or  for  specific  services  by  individual  users. 
Waiting  tinie  involves  specific  service  transactions, 
whereas  remote  storage,  etc.  involves  certain  items  of 
collection  or  equipment. 

Effectiveness  Measures 

Effectiveness  measures  include: 

•  The  amount  of  use  of  an  information  center  or  a.iy  of  its 
services  is  an  important  measure  of  effectiveness.  The 
more  a  center  or  its  services  are  used,  the  more 
effective  it  is.  The  amount  of  use  of  a  center  can  be 
measured  by  the  number  of  visits  to  it,  although  there 
are  other  forms  of  use  such  as  telephone  calls  to  a 
reference  service.  Amount  of  service  use  can  be 
measured  in  several  ways.  Collection  use  can  be 
number  of  items  loaned  or  amount  of  reading.  Uses  of 
services  should  clearly  define  and  .specify  what  is 
meant  by  use,  such  as  requests  for  reference  service,  or 
number  of  items  provided,  use  of  photocopying 
equipment  (i.e.,  an  occasion  of  use,  an  article 
photocopied  or  a  page  photocopied).  For  some 
services,  the  amount  of  use  is  the  same  as  the  amount  of 
service  output.  These  amounts  are  usually  demand- 


driven.  That  is,  the  amounts  are  largely  determined  by 
users  and  the  extent  to  which  users  will  use  these 
services  depends  a  great  deal  on  output  and  service 
attributes  such  as  quality,  timeliness,  availability  and 
accessibility.  On  the  other  hand,  the  amount  of  use  of 
some  services  such  as  access  to  materials  do  not 
correspond  to  output  quantities,  (e.g .  number  of  items 
in  the  collection). 

Just  as  with  operational  function  and  service  outputs, 
each  service  and  unit  of  use  has  associated  attnbutes 
that  can  affect  to  the  extent  of  use.  These  attributes 
include  users'  perceptions  of  service  performance, 
their  expressed  levels  of  satisfaction  with  services,  their 
indicated  importance  of  the  services  to  them,  the 
purposes  for  which  services  are  used,  and 
consequences  of  their  use.  These  attributes  are  very 
relevant  to  the  use,  usefulness,  and  value  of 
information  centers.  Additional  effectiveness 
measures  arc  as  follows: 

•  Users’ perceptions  of  services  and  service  attributes  can 
be  measured  by  their  rating  services  and  attributes. 
Service  ratings  can  be  measured  in  general  or  for 
specific  attributes  of  service.  For  example: 

General  service  performance  rating  of  reference 
services; 

Neither 

Very  Good  Veiy 

Bad  Bad  Nor  Bad  Go^  Good 

1  2  3  4  5 

Specific  attributes  such  as  relevance  of  response: 

Not  at 

All  Minimally  Very  Extremely 

Relevant  Relevant  Relevant  Relmmnt  Relevant 

1  2  3  4  5 

Some  service  attributes  are  readily  observable  and 
measures  of  users’  perceptions  are  not  always  required. 
For  example,  response  time  should  be  known  by  both 
the  center  and  the  user  and,  therefore,  measurable  by 
recording  at  the  center  and/or  reporting  by  users  (on  a 
survey  questionnaire). 

•  User  expressed  satisfaction  with  information  needs  and 
service  requirements  determine  to  a  large  degree  the 
extent  to  which  services  will  continue  to  be  used.  It  is 
difficult  for  users  to  express  satisfaction  with  how  well 
services  meet  their  information  needs  and  service 
requirements,  but  they  can  quite  easily  rate  their 
satisfaction  on  numeric  scales.  For  example,  users  can 
rate  their  satisfaction  wiih  timeliness  of  response  of  a 
reference  search  as  follows: 

Neither 

Satisfied 

Very  Nor  Very 

Dissatisifed  Dissatlsifed  Dissatisfied  Satisfied  Satisfied 
1  2  3  4  5 

Satistaclion  probably  should  be  measured  in  the 
context  of  either  specific  needs  or  specific 
requirements.  For  example,  one  should  measure 
satisfaction  with  reference  response  time  and/or  with 
relevance  of  response. 


•  Another  measure  related  to  satisfaction  is  the 
importance  of  information  needsorrequirementsorof  a 
service.  High  satisfaction  with  low  importance  for 
timeliness  has  significantly  different  meaning  (at  least 
in  terms  of  repeated  use)  than  high  satisfaction  with 
high  importance.  Importance  can  (and  should)  also  be 
measured  for  services  in  general  and  for  specific 
attributes  of  services.  Importance  can  also  be 
measured  with  rating  scales. 

Importance  of  photocopying  services; 


V#ry 

Irrytortartt 

Nor 

Very 

Unlmpoftant  Unimporttvit 

UnimporUnt 

ImporUnt  Important 

1  2 

3 

4  5 

Importance  of  amount  of  charge  for  coin-operated 
machines: 

NaWw 

Important 

Vary  Nor  Vary 

UrUmpoftant  Uittnoortant  UnImoortant  Important  ImooftanI 

12  3  4  5 

If  interested  in  comparing  across  services  an  information 
center  can  have  users  rank  services  in  order  of  their 
importance. 


One  can  rate  importance  of  levels  of  availability  such  as  loan 


period: 

Importwtc* 

ofHav^ 

Lo«i  Parted  Vary 

Mora  Than  bnportant 

important 

NaAhar 

Importartt 

Nor 

Urtimportani 

Important 

knportam 

Orta  Waak  I 

2 

3 

4 

S 
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2 

3 

4 

5 

Thraa  Waak#  1 

2 

3 

4 

$ 

regr  Waaki  i 

2 

3 

4 
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•  There  arc  many  measures  of  use  of  information  center 

materials  and 

services. 

each  of 

which  has 

some 

implication  for  the  consequences  or  value  resulting 
from  the  use  of  the  centers. 

•  We  often  characterize  purpose  of  useby  type  of  work  or 
other  function  fo-  which  the  information  or  services 
arc  used.  Such  b^/e  of  work  might  incluoc  research, 
engineering,  legal  work,  medical  care,  management, 
financial  work,  marketing  or  .sales. 

•  Consequences  of  use  can  be  expressed  in  terms  of  how 
infoiination  provided  by  information  center  or 
services  affects  users’  work  (i.e.,  input  cost,  output 
quantities,  quality,  timeliness,  etc.).  Even  through  one 
cannot  always  place  values  on  such  consequences  of 
use,  it  IS  useful  to  consider  them  and  at  least  make 
statements  about  these  consequences. 

Domain  Measures 

Domain  measures  include; 

•  Total  number  of  persons  in  service  population:  head 
counts,  FTE's 

•  Total  number  of  persons  in  user  population:  head 
counts 

•  Attributes  of  persons  in  target  and  user  populations' 
field  of  specialty  (i.e.,  chemistry,  law,  engineering, 
medicine),  work  roles  (R&D,  administration,  legal 


work,  operations,  marketing,  etc.),  age  or  years  of 
cxpenence,  degree  level. 

•  Information  behavior:  amount  of  reading,  information 
media  ised,  sources  of  information  used,  amount  of 
writirg  number  of  presentations,  number  of 
consultations,  etc. 

•  information  need:  number  of  persons  having  need  by 
type  of  information  needed  (research  results,  census 
data,  legal  briefs,  etc.),  purpose  of  use  of  information 
(see  above). 

•  Importance  of  information  as  a  resource:  rating  of 
importance  of  information  on  scales  of  1  to  4, 1  to  5,  or 
I  to  7. 

3.3  Evaluation  Models 

Evaluation  measures  by  themselves  do  not  always  provide 
sufficient  information  for  operational  decisionmaking, 
design,  planning,  and  so  on.  For  example,  knowing  the  costs 
of  services  or  the  output  quantity  is  not  nearly  as  meaningful 
to  consider  as  the  two  measures  together;  i.e.,  cost  per 
transaction  or  number  of  transactions  per  input  cost  (i  e., 
productivity). 

An  example  to  demonstrate  how  relationships  between 
measures  can  reveal  useful  management  information  is 
given  below.  In  the  example  assume  that  information  center 
staff  IS  observed  for  a  period  of  two  weeks  and  that  50  staff 
members  spend  anywhere  between  one  and  50  hours 
providing  a  service  (e.g.,  document  ordering)  or  operational 
function  (e.g.,  acquisitions).  Hypothetical  results  of  staff 
hours  worked  and  number  of  units  of  output  produced  (50 
staff  members)  are  given  below; 

EXXMPIS  OF  VALUES  FOR  DETERMINING  RELATIONSHIPS  BETWEEN 
STAFF  INPUT  AND  OUTPUT 
so  STAFF  MEMBERS  OBSERVED  OVER  TWO  WEEKS 
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The  total  number  of  hours  worked  by  these  50  staff 
members  during  the  two-wcek  period  is  1,254  hours  and 
they  produced  6,787  transactions  or  units.  Neither  of  these 
two  measures  has  much  meaning  alone,  nor  do  averages  of 
hours  per  staff  member  (25.1  hours  per  person)  or  units  per 
staff  member  (135.7  units  per  person).  However,  relating  the 
two  measures  such  as  average  number  of  units  produced  per 
hour  begins  to  have  some  useful  meaning  (5.4  units  per 
hour).  If  the  number  of  units  produced  by  the  information 
center  is  increasing  each  year  from  about  176,000  units  in 
1990  to  220,000  in  1991  the  center  must  budget  for  about 
8,148  more  hours  to  do  the  work  (i.e.,  44,000  additional 
units  divided  by  5.4  units  per  hour).  Or  if  the  average  hourly 
rate  is  $12.00,  the  budget  increase  would  be  about  $97,800 
(I.e.,  8,148  hours  times  $  12.00  per  hour). 

Tile  relationship  can  be  displayed  in  graphic  as  well  as 
tabular  form  as  shown  in  Figure  5.  Again,  the  data  displayed 
as  such  do  not  convey  a  great  deal  of  meaning.  Using  linear 
regression  the  relationship  provides  some  more  quantitative 
information  through  the  following  equation: 

y  “  a  +  bx 

--25  +  6.4X 

where  y  is  units  produced  and  x  is  number  of  hours  worked. 
This  equation  is  represented  in  the  figure  by  a  straight  line. 
Using  the  equation  one  can  approximate  or  estimate  the 
number  of  units  one  would  expect  from  a  staff  member 
working  between  1  to  50  hours  in  a  two-week  period.  For 
example,  if  one  works  30  hours,  one  would  expect  a  person 
to  produce  about  167  units.  Also,  one  can  assess  an 
individual’s  work  to  sec  whether  the  staff  member's 
productivity  is  above  the  line  (good)  or  below  it  (bad). 


)c.  cr  asJs  tons 

Fig.5  Relationship  of  number  of  hours  worked  and 
number  of  units  produced 

However,  there  is  more  information  in  the  data  than 
revealed  by  the  relationship  expressed  by  mere  average.  For 
example,  by  grouping  the  data  by  ranges  of  hours  worked  as- 
shown  in  Figure  6,  we  find  that  the  staff  are  much  more 
productive,  if  they  work  more  heavily  on  the  service. 

That  is,  the  staff  who  work  less  than  ten  hours  on  the  service 
over  two  weeks  only  produces  an  average  of  2.73  units  per 
hour.  If  they  work  1 1  to  20  hours  they  produce  an  average  of 
4.66  units  per  hour,  all  the  ivay  up  to  5.81  units  per  hour  foi 
those  who  work  41  to  50  hours.  This  relationship  (indicator) 
given  by  the  hypothetical  example  is  not  unusual  for 
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Fig.6 

information  center  services  or  operational  functions.  With 
such  knowledge,  center  managers  can  improve  productivity 
by  centralizing  services  or  batching  services.  If  this  can  be 
done  for  the  entire  year  ( 1990  in  the  example),  the  work  can 
be  done  mabout  29,830  hours  instead  of  32,600  hours,  thus 
saving  $33,240.  (i.e.,  2,770  hours  at  $12.00  per  hou  ) 

Other  factors  such  as  years  experience  can  contribute  to 
productivity  as  well.  In  the  hypothetical  example,  this  staff 
attribute  is  distinguished  by  the  two  columns  of  numbers 
above,  where  the  first  two  columns  are  experienced  staff  and 
the  second  two  columns  are  inexperienced  staff.  When 
productivity  is  calculated  by  level  of  experience  for  the 
amount  of  time  worked  it  is  found  that  experienced  staff 
average  6.02  units  per  hour  and  inexperienced  staff  average 
4.74  units  per  hour. 


AVERXOE  PBOOOCTMIY  Of  exl>E»ENCEO  (WO  INEXPERIENCED  STAFF 
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Thus,  expenenced  staff  would  appear  to  be  more  productive 
and  this  is  true  at  all  levels  of  amount  of  work.  However,  an 
important  question  is  whether  it  costs  experienced  staff 
more  per  unit  since  they  are  paid  more.  If  experienced  staff 
is  paid  $14.00  per  hour  and  inexperienced  staff  $10.00  per 
hour  the  cost  per  unit  produced  for  expeneneed  staff  is 
$2.33  per  unit  ($14.00  per  hour  divided  by  6.02  units  per 
hour)  and  the  cost  per  unit  for  inexperienced  staff  is  $2.11 
per  unit  ($10.00  per  hour  divided  by  4.74  units  per  hour). 
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Consequently,  for  this  particular  example,  it  would  cost 
slightly  less  for  inexperienced  staff  to  do  the  work.  Overall  it 
would  cost  about  $37,994  less  per  year  to  have  all  the  work 
done  by  inexperienced  staff.  The  total  cost  of  producing 
176,000  units  based  on  the  calculated  productivity  levels  by 
levels  of  experience  and  number  of  hours  worked  are  as 
follows: 


COST  TO  PROOOCC  I  ™  OOJ  UMTS  BV 
expEncNCCo  *no  HEXPcncNcro  statf 


No  d  Hour* 
WoiiwJ 

Exp«ri*nc*d 

Cxp«rtefic« 

irMiliMrwncod 

Al 

&-10 

W10S26 

6765  366 

773626 

It  10 

449605 

474  394 

463219 

21-30 

414.616 

392  667 

402206 

31  40 

364  099 

366  996 

370526 

41  SO 

391.733 

332  075 

363511 

Al 

6409  30 

6371  3C6 

391 144 

Wo  have  found  several  relationships  to  be  good  indicators 
for  evaluating  information  centers.  These  indicators  arc 
summarized  below. 


PERFORMANCe  MOtCATORS 

MEASURES 

OPERATKMAL  PERFORMANCE  INOtCATORS  (RoUM  to  Output) 

ProdociPvtty 

Output  Ouanttties/Input  Costs 

Cost  P«r  Output 

Input  Coets/Output  Ouentities 

Cc4t  By  Attfibuu  Uv«ti 

Averege  Input  Coete  By  Levels 
of  Out^  Attributes 

ProduetMty  By  AttrK>ul« 

L*v*{s 

ProductMty  By  Levels  of  Out¬ 
put  Attributes 

EFFECnVENESS  iNOtCATORS  (R«(«to  Output  to  Uso) 

Uur  Sibsfeetion 

Settsfection  Rating  Scores/ 

Number  of  Ratings 

Tuff>ov*f  R«t6 

Amount  of  Use/Output  Quantities 

Amouot  o(  6y 

Attnbut* 

Average  Amount  of  Use  By  Levels 
of  Output  Attribute 

Sati«(actKyi  by  Attribut* 

Uveii 

Average  Satisfaction  By  Levels 
of  Output  Attribute 

Amount  o<  By 

Sititf«ction  Uv«t9 

Average  Amount  of  Use  By  Levels 
of  Satisfection 

COST  EFFECTIVENESS  INOtCATORS  (Relate  Input  to  Use  of  to  Oomam) 

Cost  Per  Use 

Input  Cosis/Amount  of  Use 

Cost  Per  User 

Input  Costs/Number  of  Users 

Cost  Per  Capita 

Input  Costs/Number  In  Service 

Pop^Asfion 

Cost  By  Sebsfectlon 

Average  input  Costs  By  Levels  of  Levels 
Sabsfaction 

IMPACT  INOtCATORS  (Rotate  Use  to  Potential  Use  or  to  Domain) 

users  as  a  Proportion  of 

Population 

Number  of  Users/Number  In 

Service  Population 

Uses  Per  Capita 

Amount  of  Use/FAtmber  in 

Service  Poputetlon 

Needs  m  Ret# 

Number  of  Needs  Fined/ 

Number  of  Needs  Identified 

Amount  ot  Ubrary  U!:e 

By  Produebvity 

Amount  of  Information  Center  Use/ 

User  ProductMty 

Amount  of  Ubrary  Use 

By  Attribute  Levels 

Amount  of  Information  Center  Use/ 

User  Output  Atblbutet 

3.4  Methods 

A  listing  and  description  of  measures  were  presented  in 
Section  3.2.  The  principal  measures  include:  input  costs, 
output  quantities,  amount  of  use,  number  of  users,  user 
.satisfaction  with  services,  number  of  persons  in  the  service 
population,  user  needs  and  proportion  needs  filled.  Each  of 
these  measures  has  associated  attributes  that  can  also  be 
measured.  Several  basic  methods  can  be  used  for  obtaining 
the  measures:  internal  and  external  surveys,  resource 
allocation,  staff  records,  other  information  center  records, 
local  authority  records,  census  data,  peer  review  and  expert 
review.  Examples  of  the  methods  used  for  each  measure  arc 
presented  below: 


MEASURES 

METMOOS 

SERVICE  INPUT  COSTS 

Amount  of  resources 

Ooesive  peyments,  record  lUN  logs,  etc ,  cost  fviding  for  eiocetJon 
of  rssoucei 

SERVICE  OUTPUT 

OMnuy  Of  ou^ 

Suit  records  center  recorot  internel  surveys 

OMkty  or  output 

Observstion.  mternel  surveys,  peer  review,  expert  reVew 

Tlmefinets  of  ou^xit 

Starr  roeorde 

AveSeoiSty 

Center  records 

Aeoeii&aHy 

Staff  records,  vlesor  survey,  generel  use  survey,  tpecfftc  service 
survey,  popuetion  survey 

SERVICE  EFFECTIVENESS 

Amount  of  Use 

UOrary  records  vieiior  stMvey,  general  user  survey,  specsfo  servo# 
survey,  poputetlon  survey 

NumOMOfUiert 

Ubrery  records  visitor  survey  generel  user  survey  tpeofo  user 
Survey.  popuUtfon  survey 

User  Psreepoen  of 
ARnbuus 

Visitor  survey,  generel  user  survey  specific  servo  survey 
poputetlon  suvey 

User  Ei^ssssd 

Sstlsfsc^ 

Visitor  survey,  generel  user  survey,  epeofic  service  survey, 
popuiebon  Survey 

User  incicstsd  Importance 

Visitor  survey,  generel  user  survey.  speoAe  service  survey, 
poputstion  survey 

Purpose  of  INt 

Visitor  Survey,  generel  user  survey,  speci6d  service  survey, 
population  survey 

Consequence  of  Use 

Visilor  survey,  generel  user  survey,  specific  service  survey, 
popuietion  survey 

SEflVICe  DOMAIN 

Number  of  Persons  m 

Organizet«n  records,  teNpren#  directories. 

Poputetlon  Survey 

sajmber  of  Needs 

Population  Survey,  visitor  survey,  general  user  survey,  specific 
service  survey 

Number  ^  Needs  FVed 

Poputetlon  Suvey,  vieitor  survey,  general  user  survey,  specific 
service  eurvey 

Some  basic  concepts  of  measures  and  methods  are 
discussed  in  Parts  2  and  3.  Also  detailed  discussions  on 
measuring  staff  costs,  measuring  the  costs  of  other 
resources,  measuring  output  quantities  and  attributes,  and 
measuring  service  effectiveness  and  domain  values  are 
presented  in  those  Parts. 


Part  2 


Evaluation  of  Information 
Center  Operational 


Performance 
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Chapter  4 

Concepts  for  Evaluating  Operational  Performance 


4.1  Background 

This  chapter  provides  concepts  for  evaluating  operational 
performance.  A  description  is  also  given  of  a  system  of 
measures  that  can  be  used  by  information  center 
management  and  those  concerned  with  the  overall  budget 
and  operations  of  the  center.  In  particular,  evaluation  and 
the  system  of  measures  are  designed  to  assist  m  the  following 
functions: 

•  Monitoring  information  center  operations  for  control 
of  resources,  prevention  of  undesired  problems,  and 
diagnosis  when  trouble  occurs,  in  order  to  treat  the 
difficulty. 

•  Financial  munagemem,  including  annual  budgeting,  ad 
hoc  modifications  to  the  budget,  and  adherence  to  the 
budget. 

•  Personnel  management,  including  ( I )  determination  of 
information  center  staffing  patterns  and  (2)  periodic 
review  of  information  center  units  and  individual  staff 
members. 

•  Systems  and  equipment  management,  including 
determination  of  system  needs  and  review  of 
adherence  of  performance  to  contract  specifications 

•  Facilities  management,  including  adjustments  to 
growth. 

•  Planning,  including  setting  measurable  objectives  .and 
determining  strategies  for  change. 

•  Marketing  and  public  relations  for  achieving  objectives 
and  following  strategies. 

We  refer  back  to  the  conceptual  framework  for  evaluation  in 
Chapter  2. 

Evaluation  is  done  from  three  perspectives;  information 
center  operations,  center  users  (principally  professionals) 
and  their  work,  and  the  users’  organization.  The  reason  that 
we  suggest  that  one  focus  on  more  than  the  information 
center  operations  is  that  the  center  services  are  likely  to  have 
a  significant  effect  on  the  work  of  professionals. 
Furthermore,  the  better  these  services  are  provided  the 
greater  the  benficial  effect  on  users.  We  also  have  found  it 
useful  to  thin!:  of  the  information  center  operations  in  terms 
of  functions,  services,  activities  and  resources:  each  of  which 
has  input  costs  and  output  quantities,  quality,  timeliness,  and 
so  on.  User’s  work  processes  also  have  inputs  and  outputs 
which  are  affected  by  the  center  services.  A  system  of 
measures  can  be  derived  to  relate  the  center  outputs  to  their 
effect  on  users'  work,  at  least  through  indicators.  The 
framework  in  Figure  4  Chapter  2  shows  how  this  can  be 
done. 

In  the  figure,  the  information  center  is  shown  to  have 
functions;  each  of  which  have  inputs  and  outputs.  The  inputs 


involve  the  application  of  resources  such  as  staff,  equipment, 
facilities,  etc.,  which  are  necessary  to  perform  the  Unctions. 
We  have  identified  a  number  of  general  fiinetions  and  six 
categories  of  resources  which  are  discussed  in  detail  in  this 
Manual,  The  resource  inputs  arc  the  cost  of  resources  (e.g., 
salaries  and  wages  of  staff,  allocated  equipment  costs,  rent 
or  space  costs,  etc.)  and  the  resource  outputs  are  the  amount 
of  staff  and  their  time,  equipment  amount  and  time,  etc.  that 
are  available  to  address  the  service  needs  of  the 
organization. 

The  inputs  for  services  and  functions,  then,  are  the  total 
application  of  resources  or  total  amount  of  resources 
necessary  to  perform  the  functions  (e.g.,  S200,00  or  the  sum 
of  individual  resources  such  as  two  staff,  one  terminal,  400 
square  feet  of  facilities,  etc).  Outputs  for  services  and 
functions  arc  measured  in  such  terms  as  quantities  produced 
and  output  attributes  such  as  quality,  timeliness,  availability 
and  accessibility. 

Performance  of  staff  (or  other  resources)  can  be  measured  in 
terms  of  their  output  (e.g.,  2,000  online  searches  performed, 
average  rating  of  relevance  of  search  output  of  4.21,  average 
response  time  of  2.4  days,  3,000  hours  of  staff  available  to 
perform  online  searches,  etc.).  We  consider  such  simple 
measures  to  be  output  attributes.  Performance  can  also  be 
measured  by  relating  outputs  to  inputs.  Productivity  (which 
IS  output  quantities  divided  by  input  costs  or  amount  of  staff 
time)  is  the  most  common  such  measure.  For  example, 
productivity  might  be  0.01  searches  per  dollar  or  067 
searches  per  labor  hour.  The  inverse  of  productivity  is  easier 
to  interpret:  S 100  per  search  or  1.5  labor  hours  per  .search.  It 
is  also  possible  to  relate  input  costs  or  labor  lime  to  level  of 
quality.  For  example,  if  rating  of  quality  is  one  to  five,  (where 
one  is  the  lowest  quality  and  five  is  the  highest  quality),  one 
could  compare  average  cost  necessary  to  achieve  five  levels 
of  quality  as,  for  example; 

Level  ot  Quality  Avn.  Ckwt  Avu  Houraotlabor  Other  Coets 


One 

$45 

09 

$16 

Two 

$50 

1.0 

$20 

T>ifee 

$  60 

1.2 

$25 

Four 

$60 

1  5 

$35 

Five 

$no 

20 

$50 

Thus,  one  can  establish  how  much  it  costs  to  achieve  various 
levels  of  quality  (and  timeliness,  availability  and 
accessibility,  for  that  matter).  The  input  costs  can  vary  by 
such  factors  as  type  of  search  (e.g,,  quick  look-up  vs.  in- 
depth)  and  attributes  of  resources  (e.g.,  level  of  searcher 
knowledge,  skills,  attitudes,  terminal  baud  rate,  database 
used,  etc.)  which  affect  input  costs  and  effort  of  searchers. 

Users’  work  processes  also  have  input  resources  and  costs 
associated  with  them.  One  of  the  most  important  resources 
used  by  professionals  is  information,  which  is  often  obtained 
throu^  or  by  the  information  center.  The  cost,  amount, 
quality,  and  timeliness  of  the  information  as  a  user  resource 
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is  partially  determined  by  the  output  performance  of  the 
center.  We  have  shown  in  our  studies  that  the  extent  to  which 
the  center  services  are  used  depends  on  such  factors  as  user 
satisfaction  with  service  performance,  awareness  of 
services,  availability  of  services  (e.g.,  hours  of  opening), 
accessibility  of  services  (e.g.,  distance  to  the  center),  and 
perceived  relationship  of  price  (i.c.,  cost  to  u.se)  and  value  of 
services.  We  consider  effectiveness  of  the  information  center 
to  be  measured  in  such  terms. 

Performance  is  viewed  from  the  perspective  of  the 
information  center.  Center  management  can  control 
performance  directly  by  decisions  concerning  operations 
and  resources  and  how  they  are  allocated.  Effectiveness  is 
viewed  from  the  perspective  of  users.  However,  center 
management  can  onP'  indirectly  affect  the  effectiveness  (i.c., 
the  consequences)  o.'  center  services  because  so  many 
factors  other  than  center  performance  contribute  to 
effectiveness  (i.e.  determine  extent  to  which  services  arc 
used).  In  addition  to  output  performance,  other  factors 
which  arc  related  to  effectiveness  of  center  services  include 
extent  and  type  of  information  needs,  availability  and  cost  of 
alternative  sources  of  infoimation,  knowledge  and  attitudes 
of  users  concerning  cenlei  services  in  general,  weather,  and 
so  on.  Nevertheless,  measures  of  effectiveness  are  important 
to  observe  and  linkages  of  measures  of  input  or  output  of 
center  functions  to  effectiveness  measures  arc  useful 
indicators  and  decision-making  tools. 

Effectiveness  of  information  center  services  can  be 
measured  m  terms  of  extent  of  use,  amount  of  repealed  use, 
user  satisfaction  with  quality,  timeliness,  availability  and 
accessibility  of  services,  proportion  of  service  needs  that  are 
met  (I.C.,  so-called  fill  rates),  and  so  on.  In  addition,  one  can 
derive  a  whole  range  of  other  measures  that  provide 
indicators  of  the  performance  upd  effectiveness  of  center 
operauons  and  services 

Derived  measures  would  include: 

•  Cost  per  capita  (i.e.,  population,  potential  users,  actual 
users,  etc.), 

•  Quantities  or  use  per  capita, 

•  Average  cost  per  levels  of  satisfaction  with  quality, 
timeliness,  etc. 

•  Average  use  per  levels  of  satisfaction, 

•  Proportion  of  target  audience  reached,  and  so  on. 

Such  measures  can  be  monitored  over  time,  e.g.,  annually,  to 
provide  indicators  of  changes  in  center  operations  or  users 
served. 

We  have  shown  in  a  number  of  evaluations  that  information 
center  services  affect  user  performance  and  we  believe  that 
user  outDUts,  in  turn,  affect  the  entire  organization  served 
and  even  higher  order  effects  such  as  GNP  and  quality  of 
life.  Higher  order  effects  may  be  incorporated  in  the  mission 
of  the  organization  served  by  an  information  center,  and  the 
mission  of  the  center  is  to  help  the  organization  achieve  its 
mission,  goals  and  objectives.  The  goais  of  the  center  can  be 
thought  of  in  terms  of  the  services  provided  in  light  of  user 
information  needs  and  requirements  expressed  m  terms  of 
performance.  In  this  way,  one  can  link  the  centers  objectives 
to  goals,  and  goals  to  mission,  through  the  linkages  of 
performance  measures  to  effeetiveness  measures  to  higher 
order  effects  (even  though  such  higher  effects  cannot  be 
measured). 


A  word  needs  to  be  said  about  output  quantities  since,  for 
services  at  least,  these  quantities  can  be  the  same  as  amount 
of  use.  One  can  think  of  output  quantities  as  being 
production-driven  or  demand-driven.  Pcoduction-driven 
quantities  would  include  such  operational  outputs  as  items 
cataloged,  items  acquired,  etc.  Demand-driven  outputs  are 
those  determined  by  users'  needs,  such  as  online  searches, 
items  circulated,  amount  of  rcshclving,  etc.  In  some  ways, 
the  produclion-dnven  outputs  are  easier  to  manage  and 
productivity  based  on  them  is  more  easily  observed.  On  the 
other  hand,  if  there  is  sufficient  demand,  the  demand-driven 
output  can  be  relatively  easily  managed  as  well  Finally, 
extent  of  use  of  information  center  services  or  materials  can 
actually  be  less  than  output  quantities  (if  one  orders  but  does 
not  read  a  book,  for  example)  or  more  than  output  quantities 
(if  one  passes  on  a  book  to  colleagues,  for  example). 

In  Section  4.2  of  this  chapter,  a  list  of  possible  measures  of 
input,  output  (performance),  operational  performance, 
effectiveness  and  other  derived  measures  are  given  for 
principal  resources  and  services.  Levels  of  input,  output  and 
performance  of  information  center  operations  and  services 
are  dependent  on  a  number  of  factors.  Some  factors  are 
controllable  (at  least  largely  so)  by  management  (e.g, 
competencies  of  staff  hired,  alinbutes  of  systems  and 
equipment  purchased,  etc.),  while  other  factors  management 
can  only  indirectly  control,  that  is.  respond  to.  (e.g..  type  of 
users  served,  their  collection  requirements,  etc.). 

Theic  arc  several  types  of  resource  components  that 
management  can  utilize  to  perform  the  center  functions  and 
provide  its  services,  including. 

•  The  collection 

•  Staff 

•  Facilities  (e.g.,  space,  shelving,  seating,  etc.) 

•  Systems  and  equipment  (eg.,  computers,  terminals, 
etc.) 

•  External  services  (e  g.,coniractors,vendors.  etc.) 

There  are  attributes  associated  with  each  of  these  resource 
components  that  affect  both  input  costs  and  output 
performance  Ex.implcs  of  such  attributes  arc  listed  below 
by  type  of  resource  component' 

•  Collection 

—  Type  of  material  (books,  periodicals,  government 
documents,  microform,  A-V,  etc.) 

—  Type  of  publisher  (foreign,  domestic, 
commercial,  professional  society,  etc.) 

—  Collection  (mam,  reference,  etc ) 

—  Subject  (medical,  science,  statistics,  etc.) 

—  Category  (e.g ,  language) 

-  Age 
—  Other 

•  Staff 

—  Level  (e.g.,  professional,  paraprofession,-!!. 

clerical,  support;  GS-raling) 

—  Full-time,  part-time 
—  FI  E,  head  count,  positions  (budgeted) 

—  Levels  of  competencies  (i.e.,  knowledge,  skills, 
and  attitudes) 

—  Education 
—  Experience 

•  Facilities 

—  Total  space,  net  assignable  space 
—  Study  facilities 
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—  Shelving  (open,  closed,  remote) 

—  Handicapped  access 
—  Meeting  rooms,  etc. 

—  Utilities  (c.g.,  heat,  light,  etc.) 

—  Features  (e.g.,  architectural  design,  layout,  no.  of 
floors,  no.  of  restrooms,  etc.) 

•  Equipment 

—  Computers 
—  Staff  use,  patron  use 

—  Systems/subsystems  (e.g,  acquisitions,  serials 
control,  etc.) 

—  Features  (c.g.,  amount  of  memory,  touch 
terminals,  etc.) 

—  Photocopiers 

—  Staff,  patron  use  (free,  com-operated) 

—  Features  (e.g.,  two-sided,  color,  collation,  etc.) 

—  Audiovisual  (AV) 

—  Microform  readers/printers 

•  External  Services 
—  Contractor 

—  Vendor  .services 
—  Consultant  services 
—  Cooperative  arrangements 

These  are  examples  of  attributes  that  arc  likely  to  affect 
input  costs  and  output  performance.  Thus,  in  some 
instances  at  least,  it  will  be  important  to  observe  input, 
output  and  performance  by  the  classes  of  attnbutes.  For 
example,  copy  and  enhanced  cataloging  costs,  quantities, 
quality  and  producttvity  should  be  determined  by 
professional  and  paraprofessional  s'r''  Cost,  quantities  and 
average  cost  of  the  collection  might  be  observed  by  foreign 
vs.  domestic  publishers,  commercia’  v>  society  publishers, 
type  of  materials,  or  subject. 

There  are  three  levels  or  degrees  of  observation  of  such 
attributes. 

•  continually  observe,  present  and  analyze  data  by 
attribute  classes, 

•  continually  observe,  but  present  data  by  attribute 
classes  only  occasionally  or  for  diagnostic  purposes,  or 

•  do  not  observe,  but  have  a  means  of  collecting  data 
quickly  if  needed  for  diagnostic  or  other  special 
purposes. 

The  methods  of  observing  such  data  are  discussed  in  detail 
later. 

Finally,  some  of  the  other  derived  indicators  arc  computed 
as  per  capita  averages.  The  per  capita  computations  could 
involve  several  types  of  populations  such  as: 

Population  (per  capita  computattons) 

Entire  organization  staff  (i.e,,  all  employees) 
Professionals  in  the  organization  or  organizations 
served 

All  users  of  the  information  center 
Persons  registered  (if  a  library) 

Active  users  (e.g.,  those  who  have  used  library  at 
least  once  within  past  year) 

Visitors  to  the  center  (e.g.,  gate  counts) 

Users  of  a  specific  service 


Thus,  one  could  estimate  average  cost  per  capita  as  .average 
cost  per  center  staff  member,  per  professional,  per  user,  etc. 
Each  such  average  has  some  specific  meaning. 

Output 

Output  IS  measured  in  terms  of  quantities  produced  and 
attnbutes  such  as  quality,  timeliness,  etc.  There  are  a  number 
of  ways  m  which  work  output  can  be  measured,  although 
many  activities  simply  do  not  lend  themselves  to  specific 
output  quantities  (or  quality  or  timeliness  for  that  matter). 
Quality  can  be  observed  by  supervisors  (or  someone  else 
qualified  to  check  quality)  by  inspeettng  random  output  of 
someone’s  work.  Timeliness  can  be  observed  by  100  percent 
ob.servalion  of  events  or  with  random  spot  checks.  These 
and  other  methods  of  observing  quality  and  timeliness  are 
discussed  in  detail  in  Chapter  6. 


Not  every  activity  has  a  clear  quantity  of  output,  although 
many  do.  Even  fewer  have  quality,  timeliness,  etc.  measures. 
Table  4.1  lists  the  19  basic  functions  and  60  activities  with 
suggested  output  measures,  where  they  seem  to  apply. 

Output  measures  of  quantities  can  be  obtained  in  a  number 
of  ways,  one  of  which  is  by  individual  staff  members.  For 
example,  a  weekly  output  log  can  accompany  the  weekly 
time  log  for  .some  activities.  A  suggested  weekly  output  log  is 
given  later  in  Seelion  5.2.  On  the  log  are  output  quality 
measures  and  spaces  to  indicate  amounts  for  each  day  of  the 
week.  Supervisors  will  have  to  determtne  which  quantities  to 
collect  in  this  manner  and  then  record  the  appropnate 
output  quantity  measures  on  the  form.  The  dates  of  data 
collection  and  signatures  for  both  employees  and 
supervisors  should  be  obtained. 


4.2  Information  Center  Ferformance  and  EITectivencss 
Indicators 

In  this  section  we  discuss  measures  of  input  and  output,  and 
indicators  of  operational  and  service  performance, 
effectiveness  and  other  derived  indicators.  The  measures 
and  indicators  are  presented  initially  for  resource 
components  (collection,  staff,  systems  and  equipment,  and 
facilities)  and  then  for  principal  functions  and  services. 
Examples  of  measures  and  indicators  are  presented  along 
with  some  suggestions  as  to  the  meaning  of  them.  Below,  we 
discuss  some  important  cost  concepts  because  costs  require 
certain  rules  concerning  partitioning,  depreciation  and 
allocation  that  arc  important  to  apply  Then  resource 
indicators  are  discussed;  followed  by  examples  of  indicators 
of  operational  performance,  effectiveness,  and  services. 

In  the  previous  sections  we  briefly  discussed  input  costs  In 
this  section  we  discuss  cost  concepts  in  much  more  detail. 
Service  input  costs  are  definrd  as  the  application  of 
resources  to  provide  information  center  services.  Examples 
of  resources  include: 

•  financial  resources, 

9  collcelion, 

•  staff, 

•  facilities, 

•  equipment  and  systems,  and 

•  all  other  resources  (e.g.,  furniture,  supplies,  etc.) 
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TABLE  4.1 

OUTPUT  MEASURES 


QuwiBtlcs 


1.  Colt*ct(on  d«Mlopm*nl 

i.  ColtCttOn  tVM^Ag 


1> 

2)  ttttet  Acquired 


3 

AcquWtton* 

*  Ordtrtng 

S  PfOWMlrifl /n«i*ri,|, 
0  CUImins 


f^mawMMmwn 


•crtptteft*  ord«f«d 


f*C4K*d 


7  C«Ae«ff«|j<M« 

®  tf*L 

lUdMa  M  X.I1  . . ^ 

g^*>oq»ng 


W$%  c«n«4t(«d 

unknown 


2)  othof  lt«r«<  p<oc4M«<] 


10  Copf  ond  oftfM/teod  fitiifto/fifi  t^u. 

e««0Clna  tt<H  copy  ctithpu^O^ 

^♦c*Wog4d  4  •fthincod 

>»  *0,n-  ^ 

"  «W  .1^  c«^. 


CoUlop  Malntononco 

t»  c«^  «M»«»  ^ 


g?>y|fe4>  PfoeM.tft^ 


^7  S^*  UtktWno,  bvoodo 
**^«<Hnq^VnUno 

18,  Otttor  pny«(cU  procoMlnQ 

18.  MoooQt.ph  bfcxHfifl  mnd  f*p«fr 

20  Aoqtd«ltfon*  IM 

g»npdk*to  BkvM^j 

21,  8«|  up  of  Wndlrtfj  f#eordi 

22.  Pr*p«)na  matorUi  *«  bo  bound 

23  P^Mootinp /oiumod  bound 
volumoo 

foyofeo  Pw>y*«4i>^ 

2i.  Irrvoleo  pfopoMlng 
Briy^yKo  ondJtoodoim 
28  Otrootlonof 
26  Rofofooco 


1/  Homo  lobodoJ 
21  Iwnd  voh.  rotumod 

•«•»*»•  phyo.  pfooooood 

•♦Wsopojfod 

Bolo 

now  >ournol  titfoo 
(ool.up) 

<MU0d  to  bo  bound 
rotumod  bow»d  volumoo 

*nvo(coo  ptoco««od 

tfiroetkmol  roquoofo 
roforonco  roquooio 


Quality 

"nmelffiess 

Availability 

Accessibility 

HA 

ooopo, 

odn^,..bono)vono*o 

NA 

NA 

NA 

NA 

NA 

roquo^  tilof 

wJibdrowol 

NA 

NA 

NA 

HA 

NA 

NA 

NA 

ololmo  p«r  ofdor 

Mdor  to  rocojpt 

Womdoy*  unovoD 

NA 

HA 

NA 

tlmo 

ttonvdoy#  unovoll 

NA 

odboronco  to 
octiodulo 

8o/rHj*y#  uftovoV 

NA 

HA 

unknown 

NA 

unknown 

NA 

unknown 

NA 

unknown 

HA 

HA 

thfoughput  tlmo 
throughput  Uffio 

Horrv^ty*  unovoil 
Komdoy*  unoyoit 

NA 

NA 

%  thol  ooftform  to  otd* 

Hvoughput  timo 

ftoftvd»y*  unovoil 

NA 

*  IM  oonfomi  to  ot^ 

*  *hot  conform  to  otdo 

^  tfwi  confomi  to  $uj0 

%  ihot  conform  to  otdo 

Ibtoughput  iimo 

*hf«>wphput  Umo 

throughput  tlmo 

tf^roughput  dm# 

ftonvdoyo  unovoil 

Aorrvdoy*  unovoil 

ftorrvdoyo  ur  tvod 

NA 

NA 

NA 

NA 

NA 

occurocy  of  WtogUnput 

W 

tfwowghpu  Ifmo 

NA 

ftomvdoy*  unovoil 

NA 

NA 

NA 

««urocy  of  ItbfMna 

quobtyofprocoootog 

quoiWy  of  ropob 

HA 

throughput  t- no 
throughput  im* 

Ihteughput  tfmo 

throughput  tim* 

^•m-doyo  unovoil 
Honvdoyo  unovoj). 

hom-doyo  unovoA 

Itonvooyo  unovoil 

NA 

NA 

NA 

NA 

throughput  tlmo 

NA 

NA 

oceuroey  of  rocordo 

throughput  tlmo 

NA 

NA 

HA 

throughput  timo 

*rti«l».doyo  unovoK. 

NA 

quoMiy  of  WndJng 

throughput  timo 

ortielo-doyo  unoyoU 

NA 

OCCUrocy 

tfwoughput  timo 

NA 

na 

occuroey 

mfovonco  of  tooponoo 

NA 

1)  opood  of  roiponcf 

2)  rooponof  from 
f»*80tl*t4d  tlmo 

hour#  of  oorvico 

hour#  of  Mtvk* 

woltlng  Smo 

woWngtlm# 
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TABLE  4.1 
(CONTINUED) 


Quantities 

Quality 

Timeliness 

Availability 

Accessibility 

Oft«n*  BlbNoarADhle 

27.  Quick  look-up 

quick  lookHipa 

aocuracy  of  lodt-up 

t)  apaad  d  raaponaa 

2}  raaponaa  from 

houra  d  aarvlca 

wafting  time 

nagotialad  tima 

28  tn-d«ptfi 

ir»-dapih  aaarohaa 

1)  ralavanca  d  laaponaa 

2)  no.  ol  tema  ratrtavad 
nagodaladdma 

1)  apaad  d  raaponaa 

2)  raaponaa  from 

1)  houra  d  aatvioa 

2}  paraon  houra  ava. 

1)  waiting  time 

2)  no.  dhoat 

29  SOI 

dba  avail 

i)  proAlaa  praparad 

2}  otdpula  diatrlbutad 

1)  rafa^«rwa  ol  raaponaa 
2}  rro  ol  Hama  ratrtavad 

NA 

NA 

no.dhoatdba 

KA 

tkna  from  achadula 

NA 

tvailabla 

qA 

Ciroul^ion 

SO  Ctrculotlon 

Hama  eirculatad 

HA 

apaad  d  raaponaa 

1)  houra  d  aarvloa 

waMngtfma 

3)  aludy  aaeta/oarrak 

h  loan  period 

Shctvlfio  4nd  rWhAMnii 

Si  ShoMng  vkI  roohoMng 

Hama  ahalvad/raahalvad 

Hama  ahal^  oorractly 

throughput  time 

Harti-daya  unaval. 

32.  No  t*quHt$  luKUod  •<  libro/y 

232  raquaat  from  library 

correct  Hama  procaaaad 

apaad  d  raaponaa 

NA 

NA 

NA 

33  IntcrtltMciy  boaowing  f<«n  Hb. 

1)  Suft  procoMirvg  tt  Kbfuy 

llBa  Item  library 

correct  Hama  procaaaad 

apaad  d  raaponaa 

NA 

^  Library  procoMlng 

34  lnl«rUbr«y  borrowing  (and  no 

roquMt^  from  other  dbrarie* 

ILBa  from  othar  Uba. 

correct  Hama  procaaaad 

apaad  d  raaponaa 

NA 

NA 

39  tntfdibraiy  lorKlIng 

1}  raquaata  racaNad 

2)  U.  Hama  lulfinad 

corract  Hama  procaaaad 
correct  Hama  procaaaad 

apaad  d  raaportaa 
apaad  d  raaponat 

NA 

NA 

NA 

NA 

U*#r  Inetroctido 

36  Prepar*  axhibrta 

axhtbda  praparad 

quality  ol  axhibrta 

bma  Irom  achadula 

NA 

NA 

37.  Conduct  toura  and/or 

procont  brMnga 

toura  and/or  brMinga 

quaWy  ol  tovra/brtala 

bma  from  achadula 

NA 

NA 

30.  Conduct  trakrlng  aaaalona/ 

damonatratlona 

training  aaaalona 

quality  d  aaaalona 

drrta  from  achadula 

NA 

NA 

39  Cortduct  adtrlaory  aarvlca  aaaalona 

>  aaaalona 

quality  d  aaaalona 

tlma  from  achadula 

NA 

NA 

40.  Other  uaar  Inatrvctlon 

othar  inatructlona 

NH 

NA 

NA 

NA 

i^ocoov  Sarvleaa 

4i.  Make  photoooplara  lor  no  raquaal 

•  na  pagaa  phetocopiad 

NA 

throughput  tfma 

NA 

NA 

42.  Make  otha*’  photocoplaa 

pagaa  photocoplad 

NA 

throughput  tima 

NA 

NA 

43.  C)»<*4n  UM«  ^ 

44  Oo«n.- OPM  t  aloM 
w  maintananoa  and  aupply 

NA 

NA 

NA 

Ida  opening 

houra  upon 

NA 

waiting  tima 

NA 

Tranaladon  Sarvloaa 

45.  ht-houaa  tranaladon  ol  artlcka 
chtplara,  ale. 

1)  Hama  cl^) 

tranalalad 

2)  worda  banalalad 

•ccuridy  d  tranaldion 

1)  apaad  d  raaponaa 

2}  raaponaa  from 

NA 

NA. 

n*9<4Mad  tima 

46.  kvbouaa  archival  tranalabon 

1)  Hama  (art,  ohpg 
tranalatad 

•oouraoy  d  tranaladon 

tima  from  achadda 

NA 

NA 

2)  worda  trarw;atad 

47.  Procaaaing  and  monitoring 
Mdol-houaa  tranalatlona 

1)  Hama  (art,  cl^) 
tranalatad 

Mouracy  d  tanalation 

time  from  aehadda 

NA 

NA 

2)  worda  tr^^alatad 

48  Othar  tranalation-ralatad  activitlaa 

M 

NA 

NA 

NA 
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TABLE  4.1 
(CONTINUED) 


Quantities 

Quality 

Timeliness 

Availability 

Accessibility 

AutdfflMvd  SvM*m«  Admlni«tfatioft 

SO.  Monitor  oyitom  porformonoo 
ond  uMo* 

NA 

NA 

NA 

NA 

NA 

51.  Vondor  rotatod  odthrMoo 

NA 

NA 

NA 

NA 

NA 

52.  Stoff  rototod  MtMdM 

NA 

NA 

NA 

NA 

NA 

59.  ftopontog 

H\ 

NA 

NA 

NA 

NA 

yv  ^jifpiniwuiti) 

54.  Otiilillf  il.  oftd  Snonctol  doto 
rolatod  oclhHdM 

NA 

NA 

NA 

NA 

NA 

56.  ftoorotario^ctortool 

rolotod  oelMiiM 

NA 

NA 

NA 

NA 

NA 

95  ProtoMtonol  dovotopmoni  ortd 

troMrtg  ooiMtiM 

NA 

NA 

NA 

NA 

NA 

MoAMomonl  and  AdmMoPidon 

97.  Oortoral  admlnUtratton 

NA 

NA 

NA 

NA 

NA 

M  Ptonnino 

NA 

NA 

NA 

NA 

NA 

56.  ftnanclW  monapomont 

NA 

NA 

NA 

NA 

NA 

90  Pmonnti  marMoamant  af>d 

atafl  davatopmant 

NA 

NA 

NA 

NA 

NA 

Each  service  requires  the  application  of  one  or  more  of  the 
resources  above.  For  example,  online  database  searching 
requires  referenc'  and  support  staff,  space  for  staff  and 
service,  terminals  and  peripheral  equipment,  reference  and 
searching  materials,  and  so  on.  Generally,  these  resources 
are  quantified  as; 

•  Financial  amounts  budgeted  for  services 

•  Number  of  staff  or  staff  hours  applied  to  services 

•  Amount  of  space  allocated  to  services 

•  Number  of  equipment  items  and  systems  used  to 
provide  services 

•  Number  of  collection  items  applied  or  used 

It  is  sometimes  only  necessary  to  know  the  amount  of  staff 
time  or  equipment  time  used  for  a  service.  However,  it  is 
often  necessary  to  know  the  extent  to  which  all  resources  are 
applied  to  a  service.  Then,  the  amount  of  all  resources 
applied  to  services  can  be  converted  into  a  common  unit, 
which  IS  money  or  funds.  Once  the  amount  of  resources 
applied  to  services  is  determined,  conversion  to  measures  of 
money  or  funds  is  not  too  difficult  in  terms  of  the  following 
types  of  measures: 

•  Wages,  salaries  or  other  compensation  applied  to 
services 

•  Amount  of  space  rent  or  depreciated  expenditures 
applied  to  services 

•  Equipment  and  system  lea.se  or  depreciated 
expenditures  apphed  to  services 


•  Price  and  cost  of  processing  of  materials  and  other 
stock  items  apphed  or  used. 

It  is  useful  to  identify  attributes  of  resources  and  to  carry 
them  through  measures  of  service  input  and  output  because 
decisions  may  be  required  concerning  the  attnbutes.  For 
example,  one  may  wish  to  establish  productivity  (e  g ,  output 
quantities  divided  by  input  costs)  for  specific  levels  of  staff 
such  as  professional,  paraprofcssionals  or  support  staff.  The 
application  of  resources  needs  to  be  considered  in  a  broader 
context: 

•  services,  functions  or  activities  for  which  the  resources 
are  applied  (e.g.,  for  total  operation  of  the  information 
center,  lending  collections,  reference,  etc.) 

•  a  time  period  (e.g.,  a  year,  month,  hour,  etc.) 

The  context  for  measuring  resources  of  course  depends  on 
the  purpose  for  which  the  input  measures  will  be  used.  The 
important  thing  is  that  the  context  should  be  exactly  the 
same  for  both  input  costs  and  output  quantities  and 
attributes.  Otherwise  performanee  and  other  indicators 
cannot  be  properly  interpreted. 

A  common  time  period  for  measuring  resources  is  a  year, 
although  other  time  periods  could  have  meaning  for  specific 
problems  and  decisions  as  well.  Attributes  of  a  resource  or 
service  are  inherent  characteristics  of  a  resource  (e.g., 
education  level  of  staff)  or  of  a  service  (e.g.,  subject  of 
reference  searches.) 
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Dirtct  and  Indirect  Costs 

Methods  of  measuring  input  cannot  and  should  not  attempt 
to  be  precise  (reasonably  accurate  but  not  precise).  The 
reason  for  this  is  that  measuring  the  cost  of  resources  in  the 
context  of  a  service  often  requires  allocating  resources 
among  several  services  or  functions.  For  example,  a  staff 
member  may  spend  time  on  reference,  interlibrary  lending, 
and  administration.  Thus,  tliis  person  (or  someone  else) 
must  estimate  the  amount  of  time  spent  on  e^ch  service. 
Unless  the  person  is  observed  constantly,  it  is  necessary  to 
rely  on  memory  or  even  interpretation  of  what  one  is  doing. 
However,  most  purposes  for  which  input  costs  arc  measured 
simply  do  not  require  precise  measurement  and  the  methods 
of  measuring  proposed  here  and  elsewhere  arc  sufficient. 
The  same  qualifications  hold  for  applicalron  of  equipment, 
facilities  and  most  other  resources.  Determining  the 
application  of  all  funds  in  an  information  center  over  an 
annual  budget  period  may  require  annual  audits.  When 
audits  arc  required  (as  opposed  to  evaluation)  of  course  the 
comments  above  conci  rning  exactness  of  measures  do  not 
apply. 

Direct  cost  is  the  application  of  re.«ources  which  are 
easily  allocated  to  a  service  (e.g.,  line  charges  for  an 
online  search).  Thus,  direct  cost  is  a  cost  which  is  easily 
identified  with  a  service  (e.g.,  interlibrary  lending)  or 
resource  (e.g.,  staff). 

Indirect  costs  are  those  which  are  not  easily  assignable 
or  readily  attributable  to  any  one  service  or  function, 
activity,  or  resource.  Rosenberg'  (and  others) 
subdivides  indirect  costs  into  two  categories: 

Indirect  operating  costs.  These  costs  include  centrafiy 
budgeted  items  (e.g.,  utilities,  rent,  insurance,  etc.)  that 
are  necessary  to  the  general  operation  and 
maintenance  of  the  information  center. 

Indirect  support  costs  Costs  for  support  services  that 
benefit  overall  administration  of  the  information 
center  and  its  services  (eg,  administration, 
accounting,  personnel,  etc.) 

The  key  to  assigning  costs  into  direct  or  indirect  is  whether 
changes  in  amo  int  of  services  appreciably  affect  such  costs. 
Direct  costs  vai  v  with  changes  in  amounts  of  service,  while 
indirect  costs  dc  not.  The  indirect  costs  can  be  allocated  in 
some  equitable  way  to  direct  costs  such  as  allocating  these 
costs  in  proportion  to  amount  of  direct  costs. 

Fpted  and  Variable  Costs 

Another  concept  of  cost  finding  involves  fixed  costs,  semi¬ 
fixed  costs,  variable  costs,  and  incremental  costs.  Total  cost 
of  a  service  normally  consists  oi  ibe  application  of  a  number 
of  resources  such  as  funds,  staff,  equipment,  faci'ities,  etc. 
Sometimes  the  resources  are  applied  to  the  service  when  the 
service  is  requested  or  used.  That  is,  these  costs  vary  when 
the  service  is  used  and  are,  therefore,  called  variable  costs. 
In  other  instances,  resources  are  purchased  or  leased  for  the 
purpose  of  applying  them  to  services  over  a  period  of  time. 
However,  once  the  expense  is  incurved,  it  does  not  matter 


'  Definition  of  terms  are  adopted  from  Cost  Finding  for  Publu 
Libraries,  Philip  Rosenberg,  American  Library  Assn.,  1985. 


how  many  times  the  resource  is  used,  the  cost  is  fixed  and 
docs  not  vary.  Furthermore,  use  essentially  docs  not  deplete 
the  resource  (except  when  used  extensively).  An  example  of 
use  depleting  a  resource  is  paper  used  for  photocopies.  Tlic 
cost  of  paper  is  a  variable  cost.  Yet  the  use  of  the 
photocopier  docs  not  deplete  the  photocopier  as  a  resource 
(although  use  can  be  denied  for  a  period  of  time).  This  is  a 
fixed  cost. 

There  arc  two  kinds  of  fixed  costs:  one-time  fixed  costs  and 
recurring  fixed  costs.  One-time  fixed  costs  represent 
resources  that  are  purchased  outright  such  as  purchase  of 
books  and  equipment.  Recurring  fixed  costs  represent  costs 
that  are  fixed  for  short  time  periods  (e.g.,  a  month)  but  recur 
over  the  time  periods.  Examples  of  recurnng  fixed  costs  are 
monthly  payments  for  equipment  leases  and  facility  rellt^ 
annual  fees  paid  to  vendors  or  annual  subscription  prices 
p,!!  J  to  publishers. 

Total  cost  of  information  center  services  is  the  sum  of  fixed 
and  variable  costs  incurred  for  the  services.  Obviously  total 
cost  increases  as  the  amount  of  service  transactions  (or  units 
provided)  increases.  Figure  7  provides  an  example  of  how 
total  cost  of  a  service  increases  with  number  of  transactions 
(i.e.,  output  quantities)  for  that  service.  If  there  are  no 
transactions  Inc  total  cost  would  be  the  sum  of  fixed  costs 
applied  to  the  service  such  as  terminals,  facilities,  or  the 
collection.  If  there  are  transactions  there  would  be  an 
incremental  cost  associated  with  resources  required  to 
provide  each  transaction  (e.g.,  staff  time,  line  charges  and  hit 
rate  charges  from  the  vendor,  photocopying  of  output,  etc ). 
The  graph  shows  that  incremental  costs  are  equal  for  each 
transaction,  which  is  clearly  not  true  for  most  services. 
However,  for  theoretical  discussion  they  can  be  thought  of  .as 
being  linear.  Furthermore,  at  some  point  the  volume  of 
services  is  such  that  a  resource  is  completely  used  (e.g .  a 
terminal  is  used  to  its  capacity)  and  more  resources  must  be 
purchased  to  provide  any  additional  services  These  cosis 
arc  referred  to  as  senii-fixed  or  “step  costs"  and  must  be 
added  to  the  variable  and  fi.xed  costs  as  shown  in  the  figure 

Allocation  and  Depreciation  of  Costs 

Calculating  the  total  cost  of  a  service  would  be  relatively 
simple  if  resources  were  always  dedicated  exclusively  to 
specific  services.  Unfortunately,  this  is  rarely  the  case  excepi 
with  very  large  information  centers  where,  for  example,  staff 
can  be  assigned  exclusively  to  do  searching,  cataloging,  or 
interlibrary  loan  and  so  on.  Even  in  large  centers  computer 
equipment  and  systems  arc  used  for  multi-purposes  and 
facilities  arc  used  for  most  services  (except  those  provided  in 
branches  or  when  remote  warehousing  is  used  for  storing 
older  materials).  Allocation  of  costs  is  the  process  of 
determining  the  extent  to  which  resources  arc  applied  to 
specific  services.  For  example,  allocation  of  staff  is  done  by 
determining  how  much  time  staff  spend  on  specific  services 
and  other  activities.  Examples  of  methods  for  allocating 
staff,  collection,  equipment  and  other  resources  are 
discussed  in  the  methods  chapter  of  this  Manual  (Chapter 
5). 


^  One  could  also  think  of  staff  salancs  as  recurring  fixed  costs,  but 
there  are  reasons  for  not  doing  so,  one  of  which  is  that  personnel 
can  be  redeployed  if  this  resource  cannot  be  usefully  applied  to  a 
service. 
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TOTAL  COSTS 
(UBRAHY  SYSTEMS) 


Fig.7  Number  of  transaetions/units 


A  special  kind  uf  allocation  is  done  when  resources  arc 
expensed  rather  than  expended.  For  example,  if  equipment 
IS  purchased  outright,  the  amount  paid  is  the  expenditure. 
On  the  other  hand,  it  may  well  be  advantageous  to  allocate 
the  cost  of  a  resource  over  the  time  period  it  is  likely  to  be 
used.  This  allocation  process  is  called  depreciation.  The 
amount  allocated  is  an  expense.  Depreciation  involves 
spreading  expenditures  for  such  resources  as  computer 
equipment,  facilities  and  so  on. 

Usually  accountants  recommend  mcUiods  for  calculating 
depreciation  amounts  over  time,  or  actually  perform 
calculations  for  information  centers.  The  simplest  method 
of  calculating  depreciated  expenses  is  to  divide 
expenditures  equally  among  planning  years.  This  can  be 
done  using  the  following  steps: 

(1)  Establish  a  useful  peribd  of  life  for  the  resource;  for 
example  7  years  for  a  mainframe  computer,  5  years  for 
minicomputers,  3  years  for  microcomputer,  50  years 
for  buildings,  etc. 

(2)  Estimate  a  write-off  value  for  the  resource  at  the  end  of 
its  useful  life. 

(3)  Calculate  the  equal  annual  write-off  value;  the 
difference  divided  by  the  number  of  years  in  useful  life. 

For  example,  for  a  minicomputer: 

$12,000—51,200  $2,160  per  year 
5  years 

For  many  reasons,  accountants  prefer  not  to  use  linear 
depreciation  (i.e.,  equal  amounts  over  the  years).  One  reason 
that  linear  depreciation  is  not  used  is  that  use  of  the  resource 
Is  likely  to  reduce  over  time.  Another  reason  is  that  the  value 
of  money  changes  over  time.  Note  that  a  depreciated  cost 
can,  and  often  is,  further  allocated  among  services  (in 
addition  to  allocation  to  specific  time  period). 


Average  or  Unit  Cons 

Once  the  fixed  and  variable  cost  of  resources  have  been 
properly  allocated,  it  is  possible  to  calculate  total  cost  over  a 
specified  time  period  associated  with  a  service  or  activity. 
Average  or  unit  costs  of  services  can  be  calculated  by 
dividing  total  cost  by  total  output  quantity.  Examples  of 
average  cost  include  cost  per  book  loaned,  cost  per  online 
search,  cost  per  item  used  in  the  information  eenter,  cost  per 
interlibrary  loan,  etc.  This  derived  performance  indicator  is 
extremely  useful,  but  rarely  calculated  by  information 
centers.  Such  averages  are  probably  the  single  best  indicator 
of  how  well  a  center  is  performing  in  terms  of  costs.  The 
inverse  of  average  cost  (i.e.,  total  quantities  produced  or 
transactions  divided  by  total  cost)  is  productivity. 

An  example  is  given  below  for  calculating  average  cost. 
Assume  that  the  fixed  cost  of  a  service,  say  online  searching, 
IS  $10,000  (i.e.,  including  a  terminal,  space,  photocopier, 
furniture,  reference  materials,  etc.)  and  the  incremental  cost 
is  $100  (i.e.,  line  charges,  hit  rates,  photocopy,  staff,  etc.).  If 
there  are  two  searches  the  average  cost  is  $5,100  ($10,000  + 
2  X  $  100  divided  by  2),  three  searches  $3,433  and  so  on.  As 
the  number  of  searches  increases,  say  to  100  searches,  the 
average  cost  drops  to  $200  ($10,000 -1-100  X  $100  divided 
by  100),  a  number  much  nearer  $100,  the  incremental  cost. 
At  1,000  searches  the  average  cost  is  $  110.  One  can  continue 
to  increase  the  number  of  searches  and  the  average  cost  will 
continue  to  come  closer  to  $100  but  never  quite  reach  it. 
Thus,  the  incremental  variable  cost  is  an  asymptote  (i.e.,  an 
amount  closely  approached  but  never  reached). 

At  some  amount  of  transactions  the  average  cost 
approaches  an  asymptote  which,  in  fact,  is  the  incremental 
variable  cost  as  .shown  in  Figure  8.  This  point  at  which 
average  cost  approaches  the  incremental  cost  can  be  thought 
of  as  a  “critical  mass”.  From  an  information  center  funder's 
perspective,  it  is  desirable  for  services  to  be  provided  at  or 
near  the  critical  mass  because  it  is  at  this  point  that  the 
difference  between  costs  of  services  and  value  received  is  at 
Its  optimum.  Information  centers  and  their  branches  can 
achieve  a  critical  mass  for  such  activities  as  acquisitions  and 
physical  processmg  by  centralizing  the  functions  to  increase 
number  of  transactions  or  units  processed. 

AVERAGE  COSTS 
(LIBRARY  SYSTEMS) 


Fig, 8  Number  of  transactions/units 
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Tlie  figure  shows  that  a  “step  cost"  will  affect  average  cost, 
but  not  at  the  level  of  the  entire  step  cost  because  the  amount 
of  the  step  cost  is  averaged  over  a  large  number.  For 
example,  as.sume,  that  a  new  terminal  at  $2,OOU  is  required 
after  2,000  searches  (i.e.,  2,001  searches).  The  average  cost 
at  2,001  searches  then  is  S106  (812,000  +  2,001  X  $100  all 
divided  by  2,001),  a  value  not  much  higher  than  $105  the 
.tmount  at  2,000  searches  without  the  new  terminal. 
Obviously,  an  information  center  would  not  purchase  a 
terminal  in  order  to  conduct  one  search,  but  rather  search 
off-hours,  etc.  with  the  initial  terminal.  However,  in  theory 
and  not  too  far  off  in  practice  the  average  cost  is  as  .«''own  in 
the  figure. 

Economies  of  Scale 

$preading  of  large  fixed  costs  over  a  large  number  of 
transactions  or  units  results  in  reduced  average  cost.  This 
phenomenon  is  referred  to  as  achieving  economies  of  scale. 
Economies  of  scale  can  be  achieved  in  other  ways  as  well. 
For  example,  large  information  center  operations  can 
normally  perform  some  production-like  activities  less 
expensively  than  small  ones.  An  example  is  processing 
interlibrary  loan  requests,  indexing  or  cataloging.  By 
batching  these  activities,  tliey  can  nonnally  be  done  more 
productively  than  if  they  are  done  in  an  on-again,  off-again 
manner.  Economies  of  scale  can  also  be  achieved  in  larger 
operations  by  having  staff  with  specialized  competencies 
(i.e.,  knowledge  and  skills  developed  through  education, 
training  and  experience).  The  extreme  example  would  be 
small  information  center  run  by  a  professional  staff  member 
who  is  required  to  catalog,  search,  open  the  mail  and 
reshelve  materials.  Also,  large  operations  can  often 
negotiate  volume  discounts  for  purchasing  services, 
supplies,  materials,  equipment,  etc. 

There  are  diseconomies  of  scale  as  well.  TVpically,  .in 
information  centers  and  other  organizations  it  often  costs 
more  on  the  average  to  administer  operations  as  the  size  of 
the  organization  increases.  That  is,  the  proportion  of  total 
costs  accounted  for  by  administration  tends  to  increase  as 
the  total  costs  increase.  There  may  be  many  reasons,  if  and 
when  this  happens,  but  probably  the  principal  reason  is  that 
communication  and  maintaining  control  is  harder  in  larger 
organizations. 

Tliere  is  a  question  of  whether  one  should  use  average  cost 
(i.e.,  total  cost  divided  by  total  transactions)  or  productivity 
(i.e.,  total  transactions  divided  by  total  cost).  Each 
performance  indicator  has  a  useful  meaning.  For  example, 
funders  may  ask  what  they  are  getting  for  the  dollar 
expended.  Productivity  is  an  indicator  of  what  they  receive. 
The  equation  for  productivity  is  as  follows: 

Productivity  “  Total  transactions  divided  by  total  cost. 

This  equation  can  be  reversed  as  follows; 

Total  transactions  —  Total  cost  times  productivity. 

Thus,  if  productivity  is  increased  funders  will  get  more 
transactions  for  their  investment  or  if  they  invest  more  or 
less  they  can  forecast  tlie  effect  on  the  output.  Economies  of 
scale  play  a  role,  however,  because  of  fixed  costs.  Assume 
that  there  are  1,000  searches  and  the  fixed  and  variable  costs 
are  as  given  above.  A  funder  might  ask  what  would  hap  .en, 
if  they  cut  the  budget  in  half,  from  the  current  amount  of 
$110,000- to  $55,000.  The  number  of  searches  possible 
would  be  cut  from  1^000  to  450  (i.e.,  $55,000— $10,000  Ml 


divided  by  $100).  Thus,  the  number  of  searches  would  be 
reduced  by  more  than  one-hMf  (450  vs.  500).  If 
management  focussed  on  increasing  producHvity  of 
searchers  say,  by  25  percent  one  can  establish  wnat  effect 
that  would  have  on  cost.  If  the  $100  variable  cost,  assumes 
that  the  components  of  this  cost  is  $45  vendor  changes,  $2 
photocopying,  $3  for  support  staff  and  $50  for  professional 
time.  Then  the  totM  cost  for  1,000  searches  would  decrease 
from  $10,000  to  $97,500  and  average  cost  would  decrease 
from  $110  to  $97.50.  Thus,  savings  would  be  $12,500.  If  the 
cost  to  achieve  these  savings  is  higher  than  that  amount,  the 
decision  might  not  be  to  train,  hire  or  whatever  necessary  to 
accomplish  improved  productivity  unless  quMity,  timeliness, 
etc.  is  affected  or  that  the  productivity  improvement  affects 
vendor  costs  as  well. 

There  are  basically  four  ways  in  which  the  productivity  rates 
should  be  utilized: 

(1)  In  some  instances,  productivity  should  be  observed 
and  reviewed  on  an  ongoing  basis.  This  is  done  for 
activities  which  involve  regular  production-likc  work 
such  as  cataloging,  some  physical  processing, 
photocopying  and  so  on.  The  review  can  serve  as  a 
means  of  monitoring  individual  staff,  certain  activities 
and  units. 

(2)  Sometimes  it  is  too  difficult  to  monitor  productivity  on 
an  ongoing  basis  for  individuals  because  output  is 
difficult  to  attribute  to  a  particular  staff  member  (e.g., 
circulation)  or  the  staff  members  are  performing  a 
variety  of  activities  at  ill-dcnnable  limes.  In  these 
instances,  the  entire  activity  can  be  monitored,  but 
individuals  are  more  difficult.  Here,  it  is  still  necessaiy 
to  observe  over  time.  One  can  then  reverse  the 
calculations  and  determine  if  sufficient  output  is 
achieved  from  collective  input. 

(3)  There  are  many  activities  in  which  there  is  no 
measurable  output  such  as  in  management  and 
administrative  activities.  In  these  instances,  one  can 
still  establish,  by  the  formulas  given,  whether  or  not  the 
amount  of  time  devoted  to  the  activities  adheres  to 
norms  based  on  staffing  patterns  in  the  center. 

(4)  Finally,  as  mentioned  previously,  data  can  be  used  in 
the  aggregate  to  monitor  the  staffing  patt-  rns  and  to 
help  budget  and  plan  for  the  future  based  on  forecast 
levels  of  services. 

These  uses  of  the  productivity  measures  sho.ild  be  kept  in 
mind  when  reviewing  productivity  values. 

Other  operational  performance  would  include  the  inverse  of 
productivity  (i.e.,  hours  per  item  p.oduced  or  dollars  per 
item  produced).  In  addition,  average  cost  per  level  of  quality 
(timeliness,  availability,  or  accessibility)  can  be  computed 
from  input  and  output  measures  when  appropriate.  These 
operational  performance  measures  provide  an  indication  of 
what  it  costs  to  achieve  different  levels  of  quality.  Since 
average  cost  is  the  inverse  of  productivity  one  can  also 
derive  productivity  at  the  different  levels  of  quality.  All  of  the 
operational  performance  measures  are  derived  from  input 
and  output  measures  and,  therefore,  do  not  require  data 
collection  themselves. 

Across  the  board,  the  data  can  be  applied  in  several  ways. 
For  example,  for  upcoming  budgeting  one  can  forecast 
amount  of  activity  (e.g.,  no.  of  ILLs,  in-depth  online 
searches,  etc.)  that  should  be  expected  in  the  upcoming  year. 


By  multiplying  the  inverse  times  the  number  of  forecast 
quantities  of  output,  one  can  estimate  the  number  of  staff 
required.  For  example,  if  there  were  6,983  in-depth  online 
searches  in  the  base  year  and  that  number  is  expected  to 
grow  to  9,000.  the  staff  requirements  would  be  8,257  hours 
or  about  4.6  FTE  staff  (1  divided  by  1.09  searches  per  hour 
X  9,000  searches).  Such  computations  can  be  made  to 
establish  the  entire  information  center  staff  size  and  staffing 
patterns  can  be  determined  by  taking  the  1986  norms  and 
determining  for  each  of  ;.ie  17  functions  by  the  proportion 
that  arc  professional,  library  technician  and  clerical.  These 
proportions  then  can  be  multiplied  by  the  sum  of  staff  time 
for  all  activities  in  the  functional  areas. 

Sometimes  average  cost  is  a  better  indicator  to  utilize  than 
productivity.  First,  productivity  is  sometimes  more  difficult 
to  interpret.  For  c.xample,  if  average  cost  of  online  searching 
IS  $  100  per  search,  the  corresponding  productivity  indicator 


would  be  0.01  searches  per  dollar.  Second,  average  cost, 
when  broken  down  into  components,  can  be  used  for 
budgeting  purposes.  For  example,  if  an  information  center 
has  trend  data  on  amount  of  online  searching  done,  items 
loaned,  etc.,  it  is  oossible  to  use  a  forecast  of  amount  of 
searching  to  forecast  resouice  requirei.  ents. 

Resource  Iitdkators 

One  can  view  input  and  output  resources  in  light  of  a 
decision  to  acquire  a  resource  component  (e.g.,  hire  a  staff 
member,  purchase  a  system,  etc.).  Thus,  input  would  then  be 
the  cost  of  the  resource  component  (i.n.,  salary  and  fringe 
benefits  for  staff,  contract  amount  for  equipment,  etc.)  and 
output  would  be  amount  of  resource(s)  purchase  (i.e.,  total 
staff  hours,  number  of  terminals,  etc.)  and  expected  (or 
achieved)  quality,  timeliness,  availability,  accessibility,  and 
so  on.  Examples  of  input  and  output  are  given  in  Table  4.2, 
for  collection,  staff,  systems  and  equipment,  and  facilities. 
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For  resources,  input  is  always  measured  by  cost  in  dollars. 
For  example,  cost  of  the  collection  includes  purchase,  price, 
input  processing,  cataloging,  shelving,  storage  and 
withdrawal.  Thus,  there  are  a  number  of  resources  involved 
in  the  cost  of  the  collection  and  the  cost  will  vary  by 
collection  attributes  over  time.  The  price  paid  each  year  for 
the  collection  is  an  expenditure;  however,  we  feel  that  costs 
should  reflect  an  allocation  of  this  expenditure  over  time.  As 
an  approximation,  use  by  age  can  be  used.  Input  costs  of 
systems  and  equipment,  facilities  and  staff  should  include 
allocation  of  expenditures  and  all  other  costs  associated 
with  these  resources.  In  addition  to  attributes  of  the 
collection  given  in  the  previous  section,  another  important 
attribute  is  collection  additions  and  withdrawals,  since  they 
affect  input  cost.  Type  of  fringe  benefits  is  an  additional 
attribute  for  input  cost  of  staff. 

Output  of  resources  reflect  quantities  such  as  titles  or  items 
in  the  collection;  a  head  count,  FTEs,  hours  of  work  for  staff; 
systems  and  amount  of  equipment;  and  square  feet  or  net 
assignable  square  feet  for  facilities.  Just  as  input  cost  can  be 
classified  by  attributes,  so  can  output.  However,  it  only 
makes  sense  to  classify  both  by  the  same  attributes.  Other 
outputs  for  collection  include  the  scope  of  the  collection  in 
terms  of  what  is  covered  and  comprehensiveness  of  the 
collection  in  terms  of  how  well  it  is  covered.  Availability  of 
the  collection  is  observed  in  terms  of  (1)  the  hours  (say,  per 
week)  the  collection  is  open  and  (2)  unavailability  due  to  use 
in  the  library,  on  loan  or  missing  (misshelvcd,  stolen). 
Accessibility  is  determined  by  extent  of  open  stacks,  closed 
stacks,  or  remote  storage.  Output  of  staff  is  determined  by 
quality  of  work  and  availability  of  actual  hours  worked  (i.e., 
taking  away  vacation,  sick  leave,  holidays,  personal  time 
taken  off).  With  systems  and  equipment,  other  outputs 
resulting  from  input  costs  include  quality,  response  times 
(timeliness),  and  number  of  hours  available.  Finally,  other 
facility  outputs  include  quality  of  the  facilities,  hours  open 
(available)  and  distance  to  users  (availability). 

Indicators  of  Operational  Performance 

Indicators  of  operational  performance  involve  relationships 
between  input  costs  and  output  performance.  For  example, 
for  indicators  of  the  collection  is  operational  performance 
might  be  $2.00  pc  title  for  open  storage  and  $1.00  per  title 
for  remote  storage.  Other  examples  might  be  $20  per  FTE 
for  staff,  $20  per  net  usable  square  foot  of  space,  and  so  on. 
Average  cost  per  level  of  quality,  availability  and 
accessibility  are  indicators  of  what  it  costs  to  achieve  these 
levels.  Thus,  if  actions  are  anticipated  or  taken  to  change  the 
levels,  management  will  roughly  know  effects  on  costs. 

Indicators  of  Effectiveness 

Indicators  of  effectiveness  show  the  consequences  of  the 
resources;  input  cost  and  output  attributes.  Such  indicators 
are  generally  measured  in  terms  of  extent  of  use;  user 
satisfaction  with  quality,  timeliness,  availability  and 
accessibility;  and  some  rates  such  as  collection  fill  rates  (i.e., 
proportion  of  needs  satisfied  by  the  collection).  Extent  of 
use  of  the  collection  would  include  circulation,  in-house 
reading  and  use,  and  interlibrary  lending.  For  staff  this 
would  be  the  output  of  work  for  various  functions,  which  are 
discussed  below;  systems  and  equipment,  the  number  of 
uses  for  each  system  or  type  of  equipment;  and  visits  as  a 
measure  for  the  center.  Other  indicators  of  effectiveness 
include  number  of  users  and  proportion  of  population  who 
are  users.  One  does  not  expect  everyone  in  an  organization 


to  use  the  collection  or  visit  the  information  center  but  if 
only  10  percent  of  the  professionals  do,  or  if  the  proportion 
is  decreasing  over  time,  there  is  an  indication  of  potential 
problems.  Repeated  use  by  users  is  also  an  indicator  of 
cffectivene.ss.  Poor  service  (in  quality  or  timeliness)  can  lead 
to  use  of  alternative  services;  thus,  repeated  use  is  an 
indicator  that  users  arc  sufficiently  satisfied  to  continue 
using  the  collection,  systems  or  equipment  and  facilities. 

Purpose  of  use  and  user  satisfaction  can  be  observed 
through  user  response  forms  (foi  specific  incidences  of  use) 
or  from  general  survey  questionnaires.  Purpose  of  use  refers 
to  the  purpose  for  whicii  information  is  used,  such  as  patient 
diagnosis,  clinical  research,  management,  professional 
development,  and  so  on.  Purpose  of  use  is  an  indicator  of 
effectiveness  and  degree  to  which  the  information  center 
services  contribute  to  the  parent  organization's  mission. 
User  satisfaction  with  services  or  'ervice  output 
performance  can  be  measured  by  rating  scales  (say.  1  to  5). 
For  example,  one  can  rale  satisfaction  with  relevance  of 
online  search  results.  It  is  useful  to  obtain  such  satisfaction 
ratings  with  specifle  instances  of  use. 

Other  Derived  Indicators 

It  is  possible  to  derive  many  indicators  from  relationships 
between  input  and  effectiveness,  output  and  effectiveness, 
and  operational  performance  and  effectiveness.  Generally, 
the  following  derived  indicators  are  found  to  be  useful: 

•  Cost  per  capita  (total  population,  user  population, 
users) 

•  Cost  per  use 

•  Extent  of  use  per  capita 

•  Output  quantity  per  capita 

•  Hours  ofuse  per  capita 

•  Average  cost  per  level  of  satisfaction  —  in  order  to 
determine  what  it  costs  to  achieve  various  levels  of  user 
satisfaction 

•  Average  use  per  level  of  satisfaction  —  to  establish  the 
importance  of  user  satisfaction 

•  Purpose  ofuse  per  capita 

Other  extremely  important  indicators  arc  (1)  the  total 
demand  for  services  (i.e.,  how  much  services  arc  used  from 
the  center  and  other  sources)  and  (2)  the  costs  of  using 
alternative  sources  It  is  shown  in  Chapter  13  that  alternative 
sources  often  cost  users  much  more  in  their  time  and  m 
dollars  than  in  using  the  information  center.  The  difference 
in  cost  is  a  measure  of  the  value  of  the  center  to  users. 

Services  and  Function  Indicators 

$ome  examples  of  measures  of  input  and  output  and 
indicators  or  operational  performance  and  effectiveness  are 
given  for  services  and  functions  in  Table  4.3.  Services  and 
functions  are  different  than  resouices  in  several  ways.  First 
of  all,  input  can  be  measuied  both  in  terms  of  (1)  the  total 
cost  of  all  the  resources  necessary  to  perform  a  function  or 
provide  a  service  and  (2)  the  amounts  of  resources 
expended.  For  example,  input  to  collection  development 
can  be  dollars  expended  or  number  of  labor  hours 
expended.  Labor  hours,  of  course,  is  the  output  of  the 
resource.  Labor  hours  (and  other  amounts  of  resources)  arc 
useful  for  monitoring  productivity  of  individuals  or  an 
information  center  unit.  Yet,  ultimately,  management  may 
want  all  resources  to  be  expressed  in  the  common  unit  of 
dollars.  The  inputs  of  the  services  and  products  arc 
expressed  in  costs($)  and  amounts  <  f  resources. 
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Operational  performance  may  also  be  expressed  in  such 
terms  as  hours  per  item  produced.  However,  we  have  not 
carried  the  amounts  of  resources  to  the  other  derived 
indicators.  It  is  noted  mat  amounts  of  resources  are  also 


affected  by  attributes  and  other  factors.  Second,  services 
and  functions  are  more  likely  to  have  additional  attnbutes 
that  affect  input  and  output.  There  are  also  more  extensive 
indicators  of  effectiveness  and  other  aerived  indicators. 
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4.3  Information  Center  Input  and  Output  Measures 


1.5.  Catalog  withdrawal  activities 


In  this  section  wc  give  some  numeric  examples  of  input  and 
output  measures  and  operational  performance, 
effectiveness  and  other  derived  indicators.  These  examples 
are  given  for  19  principal  functions  (or  services)  and  a  total 
of  60  activities.  These  functions  and  activities  arc  listed 
below: 

Functions/Activities 

Collection  Development  and  Management 
1.  Collection  development 

2  Collection  weeding 

3.  Physical  withdrawal  and  related  housekeeping 

Acquisitions 

4.  Ordering 

5.  Processing  materials  received 

6.  Claiming 

7.  Cancellations 

8.  Follow-up 

Materials  Receiving  and  Mail  Processing 

9.  Matcrials/mail  processing 

Cataloging 

10.  Copy  and  enhanced  cataloging 
II  Original  cataloging 

12.  Journal  cataloging  (original  and  copy) 

13.  Added  volume  and  copy 

Catalog  Maintenance 

14.  Catalog  addidons  activities 


Physical  Processing 

16.  Spine  labelling,  barcode  labelling/'linking 

17.  Other  physical  processing 

18.  Monograph  binding  and  repair 

19.  Acquisitions  list 

Periodicals  Binding 

20.  Set  up  of  binding  records 

21.  Preparing  materials  to  be  bound 

22.  Processing  returned  bound  volumes 

Invoice  Processing 

23.  Invoice  processing 

Reference  Readers  Advisory 

24.  Directional 

25.  Reference 

Online  Bibliographic  Searching 

26.  Quick  look-up 

27.  In-depth 

28.  SDl 

Circulation 

29.  Circulation 

Shelving  and  Reshelvir- 

30.  Shelving  and  rest  c  .ng 

■  Interlibrary  Borrowing  and  Lending 

31.  Interlibrary  borrowing 

32.  Interlibrary  lending 

33.  Items  from  document  delivery  services 
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User  Instruction 

34.  Prepare  exhibits 

35.  Conduct  tours  and/or  present  briefings 

36.  Conduct  training  scssions/demonstrations 

37.  Conduct  advisory  service  sessions 

38.  Other  user  instruction 

Photocopy  Services 

39.  Make  photocopies 

40.  Check-in  users 

41.  Open  and  close,  work  at  service  desk,  provide 
maintenance  and  supply 

Translation  Services 

42.  In-house  translation  of  articles,  chapters,  etc. 

43.  Processing  and  monitoring  out-of-house  translations 

Automated  Systems  Administration 

44.  Backup  system  files 

45.  Monitor  system  performance  and  usage 

46.  Vendor  related  activities 

47.  Staff  related  activities 

48.  Reporting 

General  and  Administrative  Support 

49.  Statistical  and  financial  data  related  activities 

50.  Secretanal/clerical  related  activities 

51.  Professional  development  and  training  activities 

Management  and  Administration 

52.  General  administration 

53.  Planning 

54.  Financial  management 

55.  Personnel  management  and  staff  development 

56.  Computer,  equipment  and  systems  related  activities 

57.  Facilities  management 

58.  Contract  services 

59.  Marketing,  public  relations,  etc. 

60.  Policies  and  procedures 

There  are  at  least  500  individual  activities  that  have  been 
identified  in  information  centers.  It  is  neither  practical  nor 
necessary  to  record  the  amount  of  labor  time  devoted  to  this 
many  activities.  On  the  other  hand,  it  is  useful  to  record  the 
time  required  to  per*  ,rm  the  three  to  ten  or  so  basic 
activities  performed  by  each  employee.  The  basic  activities 
should  include  the  60  mentioned  above,  if  done  by  an 
employee.  However,  other  important  activities  should  be 
included  as  well;  particularly  if  they  are  of  intcrc.st  to  the 
employee’s  supervisor. 

One  can  structure  an  information  center  (such  as  a  library, 
clearinghouse,  etc.)  into  functions,  services  and  products  as 


outputs  of  the  functions,  activities  necessary  to  provide 
services  or  produce  the  products,  resource  components 
required  to  perform  the  activities  and  attributes  of  the 
resources,  Tliese  form  a  hierarchy  of  sorts  as  shown  in  Table 
4.4.  The  library  functions  are  sub-divided  into  technical  or 
operational  functions  (e.g.,  acquisitions,  storage,  etc.),  user 
related  functions  (c.g.,  reference,  access  to  materials, 
circulation,  etc.),  and  support  functions  (e.g.,  management, 
financial,  etc.).  The  reason  for  the  distinction  among  these 
three  types  of  functions  is  two-fold.  First,  measures  of 
output  attributes  and  effectiveness  are  different  for  them. 
Second,  costs  of  technical  or  operational  functions  are  often 
allocated  to  user  related  functions.  For  example,  it  is 
meaningful  to  allocate  costs  of  acquisitions,  document 
processing,  storage,  etc.  to  circulation  of  materials.  The 
former  being  fixed  costs  and  the  latter  variable  costs  of 
circulation  of  materials.  Furthermore,  costs  of  support 
fiinctions  might  first  be  allocated  across  both  technical  and 
user  related  functions. 

For  each  function  there  are  various  services  provided.  For 
example,  for  the  reference  functions  there  are  referral 
question  answering,  reference  search  and  Selective 
Dissemination  of  Information  (SDI).  There  are  services  that 
arc  often  accompanied  by  physical  outputs  such  as  online 
search  computer  prim-out.  The  physical  outputs  are 
products.  Within  each  service  there  are  many  activities  that 
are  performed  to  provide  the  service.  Taking  one  service,  for 
example,  reference  searching,  there  are  activities  associated 
with  solicited  searches  (c  g.,  interviewing  users,  developing  a 
search  strategy,  etc.)  and  with  unsolicited  searches.  In 
addition,  there  are  general  activities  that  must  be  performed 
to  be  able  to  do  reference  searching  (e.g.,  keeping  abreast  of 
database  sources,  reference  materials,  and  vendor  services, 
etc.).  There  arc  resources  that  are  applied  to  perform  all  of 
the  activities.  For  example,  for  conducting  searches  there  are 
staff  (people),  equipment,  communication  services, 
materials,  space  and  furniture,  and  supplies  that  are 
required.  Activities  are  what  is  done  and  resources  arc  what 
IS  necessary  to  do  it.  Each  of  these  resources  require  certain 
attributes  to  be  able  to  perform  activities  well.  For  example, 
professionals  require  certain  competencies  (e.g.,  knowledge 
and  skills  to  apply  the  knowledge).  Indicators  of  knowledge 
and  skills  are  education,  training  and  experience. 

It  IS  noted  that  every  function  has  di'^tinct  services  and 
products,  that  in  turn  have  many  activities,  requinng  several 
resources,  that  have  special  attributes.  Thus,  the  operations 
and  services  of  libraries,  clearinghouses,  and  information 
centers  are  very  complex,  but  to  properly  evaluate  them  one 
must  desenbe  them  m  the  detail  depicted  in  Table  4  4. 
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TABLE  4.4 

STRUCTURE  OF  AN  INFORMATION  CENTER 
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Chapter  5 

Evaluating  the  Performance  of  Information  Center  Staff 
and  Automated  Systems 


5.1  Background 

In  this  chapter  we  present  two  examples  of  performance 
evaluation.  The  first  example  involves  the  evaluation  of 
information  center  staff.  In  particular,  we  dLscuss  measuring 
staff  input  costs,  output  quantities  and  attributes.  Some  of 
the  examples  are  also  found  in  Keys  to  Success'.  Chapter  6 
discusses  methods  for  controlling  the  productivity  of  staff 
using  quality  assurance  surveillance  methods.  The  second 
example  discusses  the  evaluation  of  automated  systems. 
This  example  emphasizes  the  three  levels  or  perspectives 
suggested  in  the  evaluation  framework  described  in  Chapter 
2.  Some  measures  and  models  are  suggested. 


5.2.  Evaluating  Staff  Performance 
Measuring  Staff  Input  Costs 

Measuring  staff  costs  associated  with  specific  services  can 
be  difficult  unless  the  amount  of  staff  time  devoted  to 
services  is  known.  There  are  many  ways  to  measure  staff 
ume,  but  none  are  ideal.  In  fact,  different  methods  are  better 
in  different  circumstances.  Generally,  the  best  method  is  to 
allocate  staff  time  to  appropriate  services  and  operational 
functions.  The  following  rules  and  steps  are  suggested: 

•  Allocate  staff  time  only  for  the  amount  of  time  actually 
spent  working  at  the  center  (i.c.,  do  not  include 
holidays,  sick  leave,  etc.) 

•  List  all  of  the  services  or  operational  functions  of 
interest.  This  list  could  include,  for  example,  the 
functions  and  services  listed  in  Chapter  4. 

•  Design  an  individual  Weekly  Time  Log  (see  WEEKLY 
TIME  LOG  below). 

•  Design  a  worksheet  for  allocating  individual  staff  time 
and  salaries  to  services  (see  STAFF  ALLOCATION 
WORKSHEET  below) 

Three  ways  of  measuring  staff  time  arc: 

•  Use  time  sheets  or  labor  logs  on  an  ongoing  basis  of 
time  spent  on  specific  activities,  services  or  functions. 

•  Ask  staff  or  their  supervisors  to  estimate  the 
proportion  of  time  spent  on  specific  activities,  .services 
or  functions  throughout  a  year. 

•  Observe  staff  doing  various  activities  at  random  times 
during  the  day.  Random  alarm  devices  could  be  used  or 
a  supervisor  could  walk  through  the  unit  at  random 
times  and  observe  what  staff  is  doing. 

Interestingly,  the  three  ways  of  measuring  the  allocation  of 
staff  time  to  services  do  not  yield  greatly  different  re.sult.s  An 
example  of  the  use  of  time  sheets  follows. 

For  the  example,  assume  that  there  are  two  staff  members  (A 
and  B)  involved  in  Reference  and  Readers  Advisory  (R  & 
RA)  and  Online  Bibliographic  Searching  (in  addition  to 


other  services).  The  remaining  services  and  functions  and 
other  staff  are  ignored  for  the  purposes  of  the  example.  Each 
staff  member  is  asked  to  complete  a  Weekly  Time  Log  (see 
Figure  9  for  a  sample).  Assume  that  one  staff  member 
spends  1.6  hours  on  breaks,  6.2  hours  on  Reference  and 
Readers  Advisory  (directional  queries)  and  so  on.  The  staff 
member  also  spends  3.3  hours  on  Overhead  Activities  (e.g., 
discussions,  professional  development,  etc.)  and  7.2  hours 
on  Sick  Leave.  The  total  hours  spent  on  specific  services  and 
functions  can  be  estimated  for  an  entire  year  by  adding  the 
time  logged  on  52  Weekly  Time  Logs  or  a  sample  of  Time 
Logs.  In  some  instances  the  Weekly  Time  Logs  can  be  kept 
for  sampled  periods  and  extrapolated  to  a  full  year. 

On  the  Staff  Allocation  Worksheet  (see  Figure  10),  services 
and  salaries  of  specific  staff  member  (A  and  B)  are  listed. 
The  worksheets  can  be  designed  for  all  staff  or  subdivided 
by  specific  units  or  levels  of  staff  depending  on  the  size  of  the 
information  center  staff  The  salaries  can,  and  probably 
should,  include  any  fringe  benefits  or  other  forms  of 
compensation  Recorded  on  the  sample  Staff  Allocation 
Worksheet  for  each  staff  member  are  (1)  hours  worked,  (2) 
proportion  of  hours  worked  on  specific  services,  and  (3) 
appropriate  allocation  of  salaries.  For  example,  there  arc 
about  234  days  or  1,872  actual  work  hours  per  year 
(excluding  vacation,  sick  leave,  and  holidays) 

In  the  example,  staff  member  A  has  42  days  of  sick  leave  and 
holidays,  5  days  of  professional  development,  about  2  days 
of  breaks,  and  about  19  days  of  other  overhead  ume  for  a 
total  of  68  days  (490  hours)  of  overhead  time.  This  overhead 
time  can  be  recorded  (as  an  indicator  of  efficiency)  or  it  can 
be  distributed  across  the  other  services  and  functions.  The 
calculations  below  demonstrate  both  methods 

Total  staff  costs  for  services  are  calculated  using  the 
following  steps’ 

•  The  proportion  of  hours  devoted  to  each  service  is 
calculated  by  dividing  the  number  of  hours  for  each 
service  by  the  total  hours  (e.g.,  72  divided  by  1,872  “ 
0.038  or  3.8%). 

•  The  staff  cost  (for  staff  member  A)  is  caleulated  by 
multiplying  the  total  compensation  by  the  proportion 
of  hours  (e.g.,  $30,000  times  0.038  -  $1,140). 

•  The  total  staff  cost  is  found  by  adding  the  costs  across 
staff  members  (e.g ,  $1,140+S3,296“S4,436). 

The  overhead  can  be  allocated  across  services  as  follows: 

•  Calculate  the  cost  rate  attributable  to  overhead: 
($30,000)  divided  by  ($30,000-37,860)  -  i.355  for 
A  and  ($16,000)  divided  by  ($16,000-$3,520)  - 
1.282  for  B. 

•  Multiply  staff  eost  of  eaeh  serviee  by  1.355  (e.g.,  $1,140 
times  1.355  “  $1,545  for  A  and  $3,296  X  1.282  - 
$4,225  for  B). 
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1)  TO  BE  LOGGED  IN  0.1  HOUR  INCREMENTS;  EG.,  0.2,  5.8,  5.S,  ETC. 

2)  A  NORMAL  WORKDAY  IS  7.2  HOURS. 


Service/Functlone 

Mon. 

Tuee. 

Wed. 

Thura. 

Fri. 

SaL/ 

Sun. 

Total 

1.  Breake 

0.4 

0.4 

03  V 

0.5 

1.6 

2.  R&RA; 

Directional 

2.1 

1.8 

1.3 

1.0 

3.  R&RA: 

Reference 

0.5 

0.6 

0.1 

0.2 

1.4 

4.  Circulation 

31 

4,3 

1.5 

1.4 

10.3 

S.  Shelving  and 

RathaMng 

2.0 

20 

6.  Photocopy 

Senricea 

7. 

a 

— 

— 

4.0 

— 

— 

— 

4.0 

9.  Overhead 

1.1 

01 

_ _ 

2.1 

- 

MBA 

3.3 

10.  Other  (Specify) 

11.  HoUSL 

— 

— 

— 

— 

72  _ 

7.2 

TOTAL 

72 

7.2 

7.2 

7.2 

72 

36.0 

hr$. 

hr*. 

hr*. 

hr*. 

hr*. 

hia.  hre. 

hr*. 

Mo.nth 

Day 

Month 

Day 

D«t«»:  Monday  _  _  to  Sunday  _  _ ,  1990 

Employee  Signature  _ 

Supenriaor  Signature  _ 


Fig.9  Example  ot  a  weekly  time  log 
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PERIOD  OF  OBSERVATION: 


A 

B 

Spaclal/ 

Oparatlonai 

Houra 

Prop,  of 
Houra 

Comp 

Hours 

Prop,  of 
Houra 

Comp 

Total 

Compenaation 

Functiona 

1,872 

W 

$30,000 

1,872 

W 

$16,000 

$46,000 

stock  Oovciopmcnt  and 
Management 

AcquIaHlona 
Mail  Procaaaing 
Cataloguing 
Catalog  Maintananea 
Phyalcal  Procaaaing 
Parlodicala  Binding 
Invoica  Procaaaing 
Ralaranca  &  Raadara  Adviaoty 


Directional 

72 

3.8 

1,140 

395 

20.6 

3,296 

4,436 

Rafaranea 

299 

16.0 

4,800 

298 

15.9 

1,544 

7,344 

Online  Bibliographic  Searching 

Oulek  Look-up 

102 

5.4 

1,620 

1,620 

In-dapth 

391 

20.9 

6,270 

6,270 

Circulation 

87 

4.6 

1,380 

521 

27.8 

4,448 

5,628 

Shelving  &  Reahalving 

104 

5.6 

896 

896 

Photocopying  Services 
Equipment  Services 

Public  Relatione 

152 

8.1 

1,296 

1,296 

Management  and 
Administration 

431 

23.6 

6,930 

6,930 

Overhead 

490 

26.2 

7,860 

412 

220 

3,520 

11,380 

Fig  10  Example  of  a  staff  allocation  v/orksheet 
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•  Add  new  staff  costs  (e.g.,  S5,770  for  Directional  Query 

Service). 

To  determine  staff  time  and  costs  it  is  suggested  that 
supervisors  establish  about  three  to  ten  basic  activities  for 
each  staff  member  to  be  included  on  the  log  sheet.  A 
combined  log  (i.e.,  a  log  that  combines  input  time  and  output 
quantities)  can  also  be  used,  if  preferred.  The  activities  given 
on  the  Weekly  Time  Log  (see  Figure  9)  sheet  are  presented 
as  examples.  Items  2—9  could  be  left  blank  for  supervisors 
to  fill  out  for  each  staff  member.  Supervisors  should 
determine  which  activities  to  record  for  each  of  their  staff 
members.  Generally,  all  of  the  60  activities  listed  in  Chapter 
4  should  be  covered  at  minimum.  Other  activities  to  be 
included  should  be  those  that  involve  substantial  time  or  that 
the  supervisor  is  interested  in  monitoring  over  time.  The 
amount  of  time  devoted  to  various  activities  can  be 
monitored  over  time  to  establish  trends  in  the  way  in  which 
staff  members  devote  their  time,  particularly  for  activities 
that  do  not  deal  directly  with  specific  user  services  (e.g., 
acquisition  or  document  ordering)  or  those  related  to 
specific  output  (e.g.,  number  of  items  originally  catalogued). 
Otherwise,  the  input  measures  will  be  related  to  output 
measures  to  establish  operational  performance  indicators. 

The  Weekly  Time  Log  should  be  completed  each  day  to  the 
staff  members'  best  abilities.  Each  day  should  include  7.2 
hours  for  full-time  staff.  Time  should  be  recorded  to  the 
nearest  0.1  hour  to  add  up  to  7.2  hours  each  day.  Each  week 
staff  members  should  hand  in  a  log  that  they  have  signed  or 
initialled  (at  the  bottom  of  the  form)  If  absent,  the  staff 
members  should  complete  the  logs  on  their  return.  Staff 
should  also  record  the  total  amount  of  time  for  the  week  for 
each  activity  worked  on  during  the  week  (i.e.,  row  totals). 
Staff  costs  will  be  determined  by  multiplying  individual  staff 
hours  times  staff  rates  (i.e.,  salary  plus  overhead  divided  by 
1,872.)  This  can  easily  be  done  if  a  spreadsheet  program  is 
available. 


All  appropriate  resources  should  be  identified  for  each 
activity  and  cost  to  be  measured  for  resources  and  staff 
activities.  Suggestions  for  recording  these  data  arc  given  in 
the  following  example.  The  sample  spreadsheet  format  in 
Figure  11  can  be  used  as  a  guide. 

Measuring  Costs  of  Other  Resources 

Cost  finding  is  more  difficult  for  non-staff  resources.  Many 
times  it  is  best  to  estimate  the  cost  of  an  activity  on  an  itcm- 
to-item  basis  (e.g.,  total  vendor  invoices  for  online  biblio¬ 
graphic  searches;  cost  of  photocopies,  etc.  for  online 
searching).  Other  times  the  costs  of  some  resources  (e.g , 
computer  system,  postage,  some  supplies,  etc.)  can  be 
established  for  services  by  allocating  their  total  costs  across 
service  items.  Such  '■•'"ource  costs  can  be  derived  from  line 
Items  in  the  inforr  i.moi  center's  budget. 

Fixed  costs  may  require  depreciating  expenditures  over 
time.  It  may  also  be  necessary  to  allocate  the  depreciated 
costs  to  activities.  Depreciation  involves  spreading  capital 
expenditures  for  such  items  as  computer  systems, 
microform  readcr/viewers,  other  equipment,  facilities,  etc. 
over  a  useful  period  of  life.  If  equipment  and  other  similar 
expenditures  have  not  been  depreciated,  the  steps 
mentioned  above  provide  a  method  for  doing  so. 

Once  a  resource  item's  cost  for  a  year  is  determined,  it  may 
be  necessary  to  allocate  the  cost  across  two  or  more 
activities,  since  expenses  for  some  resources  or  items  might 
apply  to  more  than  one  activity.  Examples  include 
depreciated  computer  system  costs  (e.g.,  $4,0C0  per  year), 
annual  fees,  total  postage  costs,  etc.  In  these  instances,  the 
costs  must  be  allocated  between  vanable  and  fixed  costs. 
Suggested  steps  for  making  such  allocations  are' 

•  Design  an  allocation  worksheet  (see  Figure  12 
OTHER  RESOURCE  ALLOCATION  WORK¬ 
SHEET). 
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1 

t$  8,809 

t  1  1 

(Mane  | 

1 

1  2 

i  462 

1$  3,615  iHaintcnance 

1$  656 

1  1  1 

1 

i  3 
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1  1  1 
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I 
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$30,400 


$30,400 
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Fig.1l  Spreadsheet  for  displaying  direct  input  costs  of 
activities 
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Fig  12  Resource  allocation  wrksheet 
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•  For  each  cost  item  (i.e.,  resource)  establish  a 
reasonable  basis  for  determining  the  proportion  of 
costs  that  should  be  allocated  to  each  service.  For 
example,  computer  depreciation  costs  can  be  allocated 
on  the  basis  of  relative  time  used  for  services,  postage 
can  be  allocated  by  number  of  items  sent  for  the 
services,  etc. 

•  All  computer  time  can  be  allocated,  or  if  idle  time  is 
known,  the  idle  time  can  be  allocated  as  well, 
depending  on  one's  ability  to  determine  proportion  of 
time  (see  the  previous  section  on  allocation  of  staff 
overhead  cost). 

•  Proportion  of  costs  for  each  cost  item  should  be 
calculated  and  recorded. 

•  The  costs  of  each  .service  can  be  estimated  by 
multiplying  the  total  costs  times  the  proportion  (e.g., 
0.70  X  $4,000  ”  $2,800  for  computer). 

•  Total  other  resources  cost  associated  with  a  service  can 
be  found  by  adding  across  cost  items  for  each  service. 

Costs  expressed  m  terms  of  monetary  units  (dollars)  provide 
a  common  unit  of  input  for  all  resources  (and  services). 

A  sample  spreadsheet  for  displaying  input  costs  and  output 
quantities  was  given  earlier  in  Figure  II.  The  service  in  the 
example  on  the  sample  spreadsheet  is  Making  Photocopies, 
and  the  resources  used  for  the  service  include  staff, 
photocopying  equipment,  facilities,  supplies,  and  the 
collection.  Allocation  of  collection  costs  is  not  normally 
done,  but  there  are  philosophical  reasons  for  doing  so  if 
desired.  The  total  cost  across  all  resources  for  this  service  is 
$71,839.  Cost  per  document  photocopied  is  $0.90  per 
document  copy  and  cost  per  page  is  $0,098  per  page. 
Methods  for  measuring  output  quantities  are  discussed  in 
the  next  subsection. 

The  costs  of  all  resources  are  affected  by  the  attributes  of 
resources  (e.g.,  the  experience  of  staff,  the  storage  capacity 
of  computers,  etc.).  The  spreadsheet  program  should  record 
important  attributes  such  as  the  staff  level  (i.c.,  professional, 
paraprofessional,  clerical  support,  etc.)  and  salary  scale.  In 
this  way,  staff  productivity  can  be  observed  at  several  levels: 
individual  staff  productivity,  productivity  of  each  level  of 
staff,  productivity  of  all  staff  in  the  unit  (department/group, 
etc.).  It  is  nut  necessary  to  measure  the  attributes  of  other 
resources  on  an  ongoing  basis  unless  current  decisions 
regarding  such  attributes  are  to  be  made.  If,  for  example,  a 
decision  to  purchase  new  equipment  is  required,  then  one 
can  establish  the  effects  of  the  various  attributes  of  current 
computer  systems,  microcomputers,  terminals,  photo¬ 
copiers,  facilities,  etc.  on  the  service  outputs. 

Measuring  Service  Output  Quantities  and  Attributes 

Service  output  is  measured  in  terms  of  quantities  produced, 
quality,  timeliness,  etc.  There  are  a  number  of  ways  in  which 
work  output  can  be  measured,  although  many  activities 
simply  to  do  not  lend  themselves  to  specific  output 
quantities  (nor  to  quality  or  timeliness  for  that  matter). 
Quality  can  be  observed  by  supervisors’  (or  someone  else 
qualified  to  check  quality)  random  inspection  of  output  of 
someone's  work  Timeliness  can  be  observed  by  100  percent 
observation  of  events  or  by  random  spot  checks.  Many  of  the 
output  quantities  should  be  measured  as  a  matter  of  course. 
Materials  purchased,  interlibrary  loans,  reference  inquiries 
handled,  etc.,  should  all  be  quantities  that  are  kept  routinely. 
Table  4.1  in  Chapter  4  listed  basic  services  and  activities  of 


an  information  center  together  with  suggestions  for  output 
measures. 

Output  quantities  can  be  measured  by  individual  staff 
members.  For  example,  a  weekly  output  log  can  accompany 
the  weekly  time  log  for  some  activities.  A  sample  weekly 
output  log  can  be  found  in  Figure  13.  On  the  log  are  output 
quantity  measures  and  blank  spaces  for  recording  the 
amounts  for  each  day  of  the  week.  Supervisors  will  have  to 
determine  which  output  quantities  to  collect  iathis  manner 
and  then  record  the  appropriate  output  quantity  measures 
on  the  form.  The  dates  of  data  collection  and  signature  for 
both  employees  and  supervisors  should  be  obtained. 

There  is  some  merit  in  recording  output  quantities  along 
with  input  times  so  that  employees  get  a  sense  of  their  own 
productivity,  especially  if  recorded  on  a  daily  basis  as 
suggested.  It  can  be  cye-opening  for  employees  to  have  to 
record  these  data. 

Service  output  attributes  include  quality,  timeliness, 
availability  and  accessibility  of  services  and  operational 
functions.  Service  output  attributes  are  generally  more 
difficult  to  measure  than  service  output  quantities.  Methods 
are  described  in  Chapter  6  for  measuring  service  output 
attributes. 

5.3  General  Approach  to  Evaluating  Information  Center 
Automated  Systems 

The  approach  discussed  m  this  section  relates  the  evaluation 
of  an  automated  system  to  its  role  in  supporting  the  mission, 
goals  and  objectives  of  the  center,  and  therefore,  the  center's 
parent  organization  as  a  whole.  Our  approach  to  the 
evaluation  of  automated  systems  involves  three  levels  (or 
perspectives); 

•  Automated  system  level 

•  Information  center 

•  Organization  level 

The  object  of  evaluation  in  this  example  is  the  automated 

system.  However,  we  have  found  it  very  important  to 
consider  the  total  environment  within  which  an  automated 
system  operates;  otherwise,  one  might  make  decisions 
concerning  automated  systems  without  regard  to  how  the 
system  actually  affects  the  information  center,  its  users,  and 
the  organization  being  served.  Furthermore,  the  objective  of 
developing  a  methodology  to  address  future  directions  for 
the  center's  program  efforts  can  not  be  properly  achieved 
without  evidence  concerning  all  three  levels. 

Below  we  describe  in  more  detail  what  is  meant  by  the  three 
levels  of  evaluation  and  why  evaluation  is  performed  m  this 
way.  Later  in  this  section  we  desenbe  how  the  various 
measures  and  models  can  ensure  that  the  automated  system 
helps  the  center  fulfill  its  role  within  the  organization. 

Evaluation  from  the  Automated  System  Level  Perspective 

The  automated  system  can  be  evaluated  from  a  series  of 
interconnected  perspectives.  First,  the  system  can  be 
evaluated  in  isolation  from  its  environment.  This  type  of 
evaluation  is  done  in  “benchmarking”  different  automated 
systems'  performance  against  some  standard  measure  of 
performance.  In  these  cases,  each  system  being  evaluated 
against  the  benchmark  is  made  to  operate  under  the  same 
environmental  conditions  and  constraints.  The  automated 
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systems  arc  generally  “emptied”  of  all  unnecessary  data  and 
programs,  except  those  that  are  needed  to  conduct  the 
evaluation.  A  predetermined  number  of  data  records  in  a 
predetermined  format  arc  usually  loaded,  together  with  a 
program  that  will  perform  one  or  more  operations  on  the 
data.  The  program  is  then  run  for  either  a  pred.,  ‘ined  time 
period  or  for  the  time  necessary  to  perform  a  function  (in 
which  case  the  time  taken  to  perform  those  operations  is 
measured).  In  cither  case,  the  performance  of  the  system  is 
measured  m  terms  of  "quantities  produced  per  time 
interval". 

The  types  of  evaluations  range  from  the  very  simple  such  as 
number  of  additions  performed  per  minute,  or  lime  taken  to 
sort  a  list  of  N  items;  to  the  rather  sophisticated,  such  as  the 
number  of  predetermined,  simultaneous  user  tran.sactions 
that  can  be  performed  in  an  hour.  Nevertheless,  what 
distinguishes  this  level  of  evaluation  from  another  level  is 
that  the  system  is  evaluated  against  other  systems 
independently  of  the  environment  in  which  they  arc 
operated.  Such  evaluation  is  useful  for  the  following 
purposes; 

•  evaluating  the  “compute  power"  of  an  automated 
system  against  other  systems  (this  can  include  other 
systems  of  the  same  manufacturer  or  vendor,  or  other 
systems  performing  similar  functions) 

•  evaluating  whether  or  not  the  computer  configuration 
being  evaluated  can  cope  with  anticipated  workloads 
(at  both  normal  and  peak  operatirm) 

•  providing  a  baseline  of  performance  against  which 
various  operational  mixes  can  be  evaluated  (a  nearly 
“empty”  machine  being  as  close  to  an  “ideal” 
environment  as  possible). 

In  addition  to  such  automated  system  performance 
measures  mentioned  above,  one  can  also  look  at  other 
automated  system  components  as  well,  such  as  the 
information  content  or  form.  These  components  akso  affect 
system  use,  effectiveness  and  higher  order  effects. 

Evaluation  From  the  Information  Center  Perspective 

Another  level  of  automated  system  evaluation  is  from  the 
perspective  of  the  organization  within  which  it  operates.  At 
this  level  the  evaluator  would  be  concerned  with  how  well 
the  system  supports  both  the  day-to-day  operation  of  the 
information  center  and  the  needs  of  users  of  the  center.  Both 
the  center  staff  and  center  users  are  users  of  the  automated 
system.  Some  '-ystem  functions  (or  modules)  arc  used  only 
by  center  personnel  (such  as  cataloging  or  circulation), 
whereas  others  are  used  by  both  staff  and  users  (such  as  an 
online  catalog).  Thus,  an  evaluation  would  be  conducted  at 
this  level  according  to  the  specific  function/module,  and 
according  to  the  type  of  user  (i.e.,  staff  or  user)!  The 
evaluation  at  this  level  relates  how  well  the  system  performs 
in  supporting  the  particular  function  and' how  effective  the 
system  is  to  its  users.  We  think  of  performance  in  such  terms 
as  output  quantities  produced,  quality  and  timeliness.  For 
example,  the  performance  of  an  automated  cataloging 
system  could  be  measured  in  terms  of  the  time  taken  to  input 
a  complete  bibliographic  record,  or.the  time  taken  to  input 
an  itetpyhoidings  statement,  or  the  number  of  records  diet 
could  be  input  per  hour,  response  time,  etc.  On  the  other 
hand,  effectiveness  is  measured  from  the  perspective  of  the 
user  of  the  system  in  such  terms  as  amount  or  frequency  of 
use,  amount  of  repeated  use,  user  satisfaction,  etc.  Other 
factors  that  relate  to  effectiveness  include  ease  of  use  of  both 


procedures  and  equipment,  physical  comfort  of  the 
workstation/terminal  and  display,  availability  of  system  help 
messages,  availability  of  other  forms  of  assistance  (such  as 
center  staff  assistance  to  center  users),  etc. 

Evaiuation  Front  the  Organizationai  Levei  Perspective 

Often  the  mission  of  an  information  center  is  to  provide 
comprehensive  support  to  the  parent  organization  to 
achieve  its  mission.  To  any  extent  that  an  automated  system 
helps  the  information  center  to  fulfill  the  mission,  it  also 
helps  the  organization  to  achieve  its  more  global  mission. 
Thus,  this  third  level  of  evaluation  would  relate  the  use  of  the 
automated  system  to  the  use  of  the  center,  and  the  use  of  the 
center  to  the  work  of  the  professionals  it  serves.  Evaluation 
studies  of  this  type  should  demonstrate  not  only  the 
immediate  effects  of  system  use  but  also  the  extent  of 
subsequent  or  higher-order  effects. 

Linking  the  Three  Leveis  of  Evaiuation 

The  general  approach  to  evaluating  the  automated  system  is 
displayed  in  Figure  14.  Across  the  top  of  the  figure  aic 
several  related  aspects: 

•  hardware/software  components 

•  machine  performance 

•  system  factors 

•  system  attributes/vanables 

•  system  effectiveness 

•  higher  order  cffects/consequences  of  system  use 

The  first  four  aspects  correspond  to  the  automated  system 
level  referred  to  previously,  the  system  effectiveness  to  the 
information  center  level  and  higher  order  effects/ 
consequences  of  system  use  to  the  organizational  level.  The 
arrows  indicate  that  one  aspect  affects  the  next  one.  That  is. 
hardware/software  components  affect  machine 
performance,  which  affects  certain  system  factors  which 
affect  system  attributes  (or  variables),  which  affect  system 
effectiveness,  which  affects  consequences  of  system  use. 

The  system  hardware/software  components  include  CPU, 
internal  memory,  external  memory,  functions/modules, 
indexing  structure,  and  communications.  The  machine 
performance  of  quantities  processed,  procc.ssing  time, 
response  time,  etc.,  clearly  depend  on  these  components  (as 
well  as  other  components  such  as  vendor).  Furthermore,  the 
functions  (or  modules)  supported  by  the  system  are  an 
important  system  factor  and  the  decree  to  which  the  overall 
system  performs  ts  dependent  on  .iiesc  functions.  System 
effectiveness  depends  on  several  system  attributes  of  the 
function/module  factor.  Such  attributes  are  the  quantities  of 
information  available,  quality  (accuracy  and  precision), 
currency  of  information,  timeliness  of  response,  cost  and  so 
on. 

We  suggest  grouping  system  factors  and  their  attnbutes 
together  because  the  information  center  has  some  direct 
influence  on  them  through  ■  ystcni  configuration  and 
management  modifications. 

In  addition  to  functions/modules  supported,  another  factor 
over  which  the  center  has  control  is  promotion,  which  i  'fccts 
user  awarene.ss,  perception  and  understanding  of  the 
system;  all  of  which  partially  contribute  to  system  use  and 
effectiveness. 
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There  are  some  system  factors  which  cannot  be  influenced 
directly  by  center  or  organization  personnel,  but  knowledge 
of  which  can  be  very  helpful  because  of  their  indirect  effect 
on  use.  These  include  user-related  and  environmental 
factors.  User-related  variables  include  the  education, 
training  and  specially  of  users,  their  exposure  to  and  use  of 
automated  systems,  and  age.  Evidence  suggests  that 
frequency  of  use  of  automated  systems  is  related  to  such 
variables.  Similarly,  environmental  factors  such  as  location 
of  terminals  and  visibility  of  terminals  to  users  also  affect 
system  use. 

One  approach  to  this  level  of  evaluation  is  to  measure  and 
evaluate  system  attributes  and  variables  and  to  develop 
models  for  estimating  the  relative  contribution  that  they 
make  to  the  use  of  the  system  and  then  to  the  consequences 
of  system  use. 

Most  evaluations  merely  measure  either  attribute;  znd  user 
satisfaction,  or  frequency  of  use.  The  power  of  models  for 
actually  linking  attributes  with  frequency  of  use  is  thai  tl  sy 
estimate  the  relative  contribution  that  attributes,  suen  as 
cost  (to  use),  timeliness,  quality,  etc.,  make  to  frequency  of 
use.  Thus,  decisions  can  be  made  concerning  functions/ 
modules  supported  and  promotion  alternatives  based  on 
both  cost  and  effectiveness  (in  terms  of  amount  of  use  of  the 
system). 

Frequency  of  use  per  se  does  not  totally  indicate  system 
effectiveness.  One  must  consider  higher  order  effects  as 
well.  Obviously,  improved  user  productivity,  better  research 
and  more  effective  management  are  important 
consequences  of  the  system.  It  is  very  useful  to  determine  the 
effectiveness  of  information  systems  by  estimating  the  cost 
and  benefits  of  systems  by  comparing  the  current  system 
with  alternative  methods  of  obtaining  information  (or 
aitematives  to  not  having  the  information).  Such 
comparisons  are  made  at  all  the  levels  mentioned  above;  that 
is,  system  attributes,  frequency  of  use,  and  consequences  of 
use.  At  each  level,  if  the  comparison  is  favorable  it  is  counted 
as  a  benefit  and  if  it  is  unfavorable  it  is  counted  as  a  cost  (or, 
perhaps  more  appropriately,  a  detriment).  As  the  levels 
aggregate  it  becomes  more  and  more  difficult  to  measure 


and/or  model  so  that  it  may  be  necessary  to  make 
comparisons  in  nor-quantifiable  terms. 

Finally,  in  this  evaluation  approach  it  is  assumed  that  the 
mission  of  the  automated  system  is  in  some  way  related  to 
the  higher  order  effccts/conscquenccs  of  the  system.  The 
goals  of  the  system  are  broad  statements  of  ways  in  w'hich  the 
mission  can  be  achieved.  System  objectives  can  be  stated  m 
terms  of  system  use  or  effectiveness.  Within  each  goal  are  a 
set  of  such  measurable  objectives.  Then,  using  the 
evaluation  approach  set  forth  in  this  chapter,  one  can  also 
make  decisions  on  the  basis  of  how  they  affect  objectives, 
goals,  and  mission  of  the  information  center. 

Deiaits  of  the  Aiilomaied  System  E"ahiation 

As  mentioned  above,  there  are  many  factors  that  contribute 
to  the  use  and  usefulness  of  a  system  such  as  an  automated 
system.  The  information  center  has  direct  control  over  some 
of  these  factors  through,  for  example,  system  design 
modifications.  With  other  use-related  factors,  the  center  can 
only  take  indirect  actions  such  as  by  training  programs. 
Below  we  sub-divide  the  system  factors  into  groupings  that 
imply  different  levels  of  solution  or  action  that  can  be  taken. 
Associated  with  each  of  these  system  factors  are  attributes 
or  variables  that  can  be  measured  and  which  can  be  used  in 
models  for  estimating  purpose  and  amount  of  use,  or 
effectiveness.  The  models  can  also  determine  the  relative 
contribution  each  attribute  makes  to  the  frequency  of  use  of 
the  system.  The  listing  of  potential  .system  components  and 
their  attributes  are  given  in  Figure  15. 

Information  is  sub-divided  into  two  principal  components: 
information  content  and  information  form.  Information 
content  is  the  acmal  message  which  is  being  conveyed. 
Attributes  related  to  information  content  include  accuracy 
and  precision  (i.e.,  the  ‘"dosage”  of  information  content:  is  it 
too  much  or  too  little?).  Other  attributes  related  to  use  and 
usefulness  are  currency,  completeness,  and  availability.  The 
second  information  component,  form,  includes  the  format 
of  the  information,  the  structure,  and  medium  (e.g., 
electronic,  paper,  microform,  etc.).  Together  the  different 
aspects  of  information  form  comprise  the  “package”  used  to 
communicate  the  information  content.  Examples  of 


f 


5 


I 


47 


Infocaation  Content  ^^cura^ 

Piecislon 
Cutcency 
Caqpletcnesfl 


Infonnation  Pom 


FOCHAt 

Stcucturo 

Mediuta 


TlMllnoM 
Access  ibUlty 
AvalUbility 


Access  n^dumism 


SArduar* 

Software 

Ccnnunications 


Re^Mnse  time 
Resource  utilization 
B&se  of  use 
PlexibUiry 


Fig.15  Relationship  of  functions/modules  supported 
factors  and  system  attributes 


attributes  of  information  form  are  given  in  the  figure  and 
include  accuracy,  precision,  timeliness,  and  arc  ^ssibility.  As 
a  matter  of  further  clariflcation,  we  also  ..link  of  the 
information  content  along  with  purpose  of  use  as  user 
information  needs,  and  the  attributes  associated  witf. 
information  content  and  form  are  user  information 
requirements.  Both  are  essential  to  evaluating  system 
effectiveness  and  value. 

In  addition  to  the  content  and  form  the  automated  system 
includes  an  access  mechanism.  One  major  co,nponcnt  of 
this  mechanism  is  hardware/software  used  to  provide  user 
accc.ss  to  the  automated  system.  This  includes  computers, 
computer  storage  units,  and  various  other  peripheral 
devices,  such  as  terminals,  printers,  modems,  and 
controllers.  The  hardware/software  employed  can  seriously 
affect  ultimate  system  performance  and  system 
effectiveness.  Oi'’  'r  hardware  considerations  affecting 
system  nerforr...  :  and  effectiveness  are  the  types  of 
communications , .  ■  ;s  supported  by  modems,  the  amount  of 
internal  memory  in  the  computer  supporting  the  database, 
etc. 


Another  component  of  the  access  mechanism  is  the 
communications  infrastructure  which  supports  user 
interaction  with  the  database.  The  nature  and  extent  of  this 
infrastructure,  measured  in  terms  of  type  of 
communications  line  (direct  dedicated,  dial  up,  voice-grade, 
data  lines,  hardware,  etc.),  can  also  affect  system 
performance  and  effectiveness.  The  speed  of  data 
communications  can  affect  the  speed  of  response  of  the 
system  (as  will  some  of  the  hardware  attributes). 

The  components,  hardware/software  and  communications, 
comi  ,ing  the  access  mechanism  should  be  evaluated  in 
terms  of  how  they  affect  system  performance  and 
effectiveness.  For  example,  output  attributes  can  be 
measured  in  response  time,  flexibility  of  the  user  interface, 
case  of  use  of  the  user  interface,  efficiency  of  system 
resources  utilization.  These  system  output  measures  can  be 
related  to  effectiveness  through  correlation  with  frequency 
of  system  use.  purpose  of  use,  user  satisfaction,  and  so  on. 
Alternatives  that  might  be  developed  to  improve 


performance  and/or  effectiveness  include  the  possible 
development  of  multiple  user  interfaces  (for  example,  one 
for  experienced  users  and  one  for  occasional  users,  or  one 
for  specialists  and  one  for  non-specialists)  to  improve 
flexibility  and  use  of  the  system. 

Promotion  of  information  systems,  services  or  products  can 
be  evaluated  in  terms  of  their  attributes,  effects  on  frequency 
of  use,  and  higher  order  effects.  Presumably,  the  objectives 
of  promotion  arc  also  to  increase  the  use  and  effectiveness  of 
the  automated  system  and  achieve  it  goals  and  mission.  One 
can  subdivide  promotion  into  public  relations  and  price  (see 
Figuie  16).  There  are  many  ways  that  public  relations  can  be 
achieved  including  exhibits  at  the  center,  flyers,  or 
newsletters  sent  to  users. 


Public  Rclattono 

Exi.ibits 

ItewslettArs 

Plv«« 

Direct  ccntacts 


Exchange  of  nenies 
Other  Ccets 
Osar  Time 


Awareness 

Perc^ion  of  system 

Understanding 

Recall 

Mmbec  of  co^t^kct8 
Ihenes 


Dollars 
labor  tijDs 


Fig.16  Relationship  of  promotion  factors  and  system 
attnbutes/variables 


Normally  price  or  cost  of  use  are  included  as  an  attribute 
(e.g.,  in  conjoint  trade-off  analysis),  but  wc  included  it  here 
as  a  component  of  promotion.  The  reason  that  wc  normally 
evaluate  price  as  an  information  service  attribute  is  that 
users  pay  for  both  the  information  content  and  information 
form  (i.c.,  value  added  by  improvements  in  attributes  of 
format,  structure,  and  medium).  Thus,  wc  have  found  it  very 
useful  to  determine  the  trade-offs  to  users  in  terms  of  what 
they  are  willing  to  pay  for  various  levels  of  quality, 
timeliness,  etc.  In  the  automated  system,  the  “price”  paid  for 
information  content  is  the  time  and  effort  users  expend  in 
gaining  access  to,  assimilating  and  using  information.  We 
have  found  with  bibliographic  information  systems,  that  the 
“price”  is  sometimes  too  high  to  pay.  Thus,  users  either  use 
intermediaries  or  simply  do  not  get  the  information  (see 
Chapter  9). 

There  are  a  number  of  user  and  environmental  factors  that 
can  be  correlated  with  purpose  of  use  and  frequency  of  use 
of  the  automated  system.  'There  are  some  user-related 
factors  that  link  information  content  and  form  to  frequency 
of  information  use.  For  example,  certain  ways  of  expressing 
meaning  varies  by  specialty  and  age.  Certainly,  experience  in 
using  various  media  is  linked  to  age.  Tltese  relationships  are 
shown  in  Figure  17. 

There  are  several  important  or  potentially  useful  user- 
related  factors  that  mi^t  be  used  in  evaluating  such  as  age 
and  resistance  to  new  technology.  We  have  also  found  that 
there  are  a  number  of  environmental  factors  that  are  related 
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to  frequency  of  use  in  systems  as  well.  The  most  important 
such  environmentui  Tretor  is  the  location  of  terminals  and, 
relatedly,  visibility  of  tlie  terminal  to  users,  distance  (in  time) 
to  the  terminal  and  availability  of  a  terminal  based  on 
competition  for  its  use. 
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Fig.17  Relationship  of  user  related  factors  and  system 
attrfbutes/variables 

Over  the  years  we  have  made  a  definitional  distinction 
between  system  performance  (e.g.,  output  quality,  quantities 
produced,  timeliness,  etc.)  and  e^ectiveness  which  measures 
the  consequences  of  the  system.  Design  and  other  system 
decisions  can  directly  affect  output  attributes,  but  one  must 
assume  that  higher  quality,  faster  response,  etc.  will  have 
positive  effects  such  as  greater  user  satisfaction,  more 
frequent  use  and  positive  higher  order  effects  such  as  use  of 
primary  materials  and  how  that  affects  work.  Later  we  will 
show  how  to  determine,  through  models,  the  extent  to  which 
output  attributes  affect  frequency  of  use.  Here  we  discuss 
examples  of  measures  of  first  order  effects  of  the  automated 
system. 

The  first  effectiveness  measure  is  user  satisfaction  (see  Part 
3).  Here  system  users  are  asked  to  indicate  their  satisfaction 
with  system  performance  such  as  with  system  response  time, 
accuracy  and  so  on.  These  measures  are  made  by  employing 
Likert  like  scales  (1  to  S).  Then,  actual  measured  attributes 
are  correlated  with  user  satisfaction.  For  example,  one 
should  be  able  to  obtain  measures  of  system  response  time 
and  user  satisfaction.  System  response  time  can  be  observed 
generally  with  the  controlled  system  enhancement  and 
experimentation  and  correlated  with  general  observations 
of  satisfaction  with  this  attribute.  However,  if  possible, 
critical  incidents  of  use  should  be  employed  to  directly 
correlate  observed  response  times  with  the  satisfaction 
indicated  in  a  user  survey. 

Frequency  of  use  is  partially  dependent  on  user  satisfaction 
However,  we  have  found  tliat  both  measures  of  effectiveness 
should  be  observed,  over  time.  One  reason  is  that  increased 
frequency  of  use  over  time  can  be  misleading,  if  for  example, 
there  is  a  great  deal  of  initial  trial  (or  learning)  of  a  system  as 
may  be  the  case  with  the  new  automated  system.  For  this 
reason,  we  also  like  to  measure  “trial  and  repeat”  use  of 
systems.  Secondly,  declining  use  does  not  tell  why  the  system 
is  being  used  less.  One  can  often  determine  why  by  analyzing 
user  satisfaction  with  specific  performance  attributes  and 
the  actual  attributes  measures. 

In  addition  to  frequency  of  use  it  is  important  to  determine 
purpose  of  use.  This  is  necessary  because  certain 
performance  attributes  may  be  important  to  specific 
purposes  of  use.  An  extreme  example  is  that  speed  of 
response,  accuracy  of  information,  content,  and  highly 
picuise  information  is  absolutely  necessary  sometimes.  Such 
levels  of  performance  attributes  are  not  necessary  at  other 
times  or  for  some  users.  Purposes  of  use  are  related  to  the 
value  that  is  derived  from  information  use. 


All  of  the  principal  measures  of  automated  system  first 
order  effectiveness  should  be  ob-served:  user  satisfaction 
with  specific  performance  attributes,  repeated  use, 
frequency  of  use  and  purpose  of  use.  Specifically  which 
performance  attributes  and  purposes  of  use  to  be  employed 
in  the  evaluation  should  be  determined  early  in  the  project. 
The  uses  can  be  sub-divided  into  meaningful  categories 
depending  on  relationship  to  both  performance  attributes 
and  likely  higher  order  effects.  Such  categorization  can  be 
determined  by  a  combination  of  prior  work  at  the  center, 
experience  with  similar  systems,  review  with  knowledgeable 
experts,  and  literature.  All  of  the  measures  chosen  for  the 
evaluation  should  be  incorporated  into  models  of  these 
relationships,  if  possible. 

The  true  justification  for  information  system  expenditures  is 
higher  order  effects  achieved  as  a  result  of  using  the  system. 
Higher  order  effects  can  be  observed  and  measured  by 
either  asking  professionals  to  indicate  what  these  effects  are, 
or  by  observing  user  behavior  over  a  period  of  time.  Either 
way,  the  estimates  are  usually  “soft”  because  of  the  many 
other  factors  that  contribute  to  user  behavior.  We  have 
found  that  observing  specific  critical  incidents  of  use 
(chosen  randomly  from  a  set  of  uses  over  a  specified  time 
period)  provide  much  better  analytical  results  than  asking 
general  questions  about  information  use  and  its 
consequences.  For  survey  purposes,  this  permits  the  users  to 
focus  on  specific  performance  attributes,  satisfaction, 
purpose  of  use,  and  consequences.  This  also  provides  a 
much  more  statistically  sensitive  correlation  of  these 
measures.  It  is  anticipated  that  such  observations  will  come 
from  the  users  as  well  as  online  surveys. 

There  are  several  powerful  models  for  linking  information 
system  attnbutes  to  frequency  of  use  (Figure  18).  Which  of 
these  models  should  be  used  for  the  automated  system 
evaluation  depends  to  some  degree  on  the  evaluation  plan. 
The  first  of  these  models  is  a  conjoint  measurement  trade¬ 
off  model.  This  model,  in  effect,  establishes  the  relative 
importance  of  system  attributes  such  as  quality,  timeliness, 
response  time,  etc.  The  conjoint  method  involves  a 
questionnaire  and  interviewing  technique  that  forces  users 
to  make  choices  between  levels  of  pairs  of  attnbutes.  Value 
“utilities”  are  derived  from  these  user  choices  to  indicate  the 
relative  importance  of  all  the  attributes  (and  their  level)  or 
compare  several  attributes  against  one  such  as  cost  to  use 
One  can  then  correlate  the  utilities  of  the  attributes  with 
frequency  of  use  (of  an  information  product).  Our  greatest 
success  with  the  conjoint  measurement  method  has  been 
with  critical  incidents  of  use  rather  than  with  general 
comparisons  of  attributes.  This  is  because  every  single  use 
varies  in  terms  of  relative  irr.portance  of  attributes  (sec 
Chapter  11). 

Examples  of  conjoint  measurement  analysis  include:  online 
and  manual  bibliographic  searching  (two  different  systems), 
products  involving  catalog  subject  authorities,  document 
delivery  systems  (i.e.,  distribution  of  journal  article  copies). 
Examples  of  the  important  attnbutes  of  these  information 
systems  are  relevance  of  output,  number  of  items  retneved, 
speed  of  delivery  and  price  (bibliographic  searching), 
frequency  of  publication,  publication  medium,  cumulative 
or  not,  and  price  (for  the  subject  authority  product),  and 
speed  of  delivery,  reproduction  quality,  availability  of 
special  graphics  or  not,  and  price  (for  the  document  delivery 
system).  In  each  iastance,  the  trade-off  analysis  can  bedone 
for  different  user  population  segments  or  for  comparison  of 
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several  attributes  against  one  specific  use.  Data  collection 
methods  included  self-administered  questionnaires  and 
telephone  interviews.  Another  kind  of  trade-off  analysis  can 
be  performed  using  disaggregated  multiple  regression 
models.  This  modelling  technique  has  the  advantage  over 
conjoint  measitrements  of  incorporating  both  product 
attributes  as  well  as  other  factors  such  as  user  and 
environmental  factors.  For  example,  for  scientific  journals, 
we  determined  the  relative  contribution  that  (I)  quality  of 
journals,  (2)  number  of  articles,  (3)  price,  (4)  whether  a 
journal  is  part  of  a  professional  society  membership  or  not, 
(5)  availability  in  information  center,  and  (6)  distance  to 
information  centers  makes  to  personal  subscription  to 
journals.  The  problem  with  this  technique  is  that  it  is  very 
expensive. 


Another  model  involves  cluster  analysis.  A  form  of  cluster 
analysis  is  market  segmentation  or  multi-dimensional 
scaling  to  determine  groups  of  users  (grouped  by  their 
atlnbutes)  that  are  more  likely  to  use  systems  such  as  the 
automated  system.  The  problem  with  the  way  researchers 
analyze  market  segments  or  clusters  is  that  these  clusters  are 
not  then  related  to  frequency  of  use.  By  using  multiple 
regression  modelling  (or  disaggregated  multiple  regression 
analysis)  three  linkages  can  be  made.  For  example,  the 
dependent  variable  can  be  frequency  of  use  and  the 
independent  vtiriablcs  can  be  system  component  attributes, 
promotion  attributes,  environmental  attributes  and  user 
related  attributes,  promotion  attributes,  environmental 
attributes  and  user  related  attribute.’  determined  from 
cluster  analysis. 
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Chapter  6 

Quality  Assurance  Surveillance  of  Information  Center  Operations 


6.1  Background 

Chapters  3, 4,  and  5  discuss  evaluation  of  the  performance 
of  information  center  operations  and  services.  In  particular, 
we  discuss  performance  evaluated  from  the  standpoint  of 
input  costs  and  attributes,  output  quantities  and  attributes, 
and  relationships  between  input  and  output.  Information 
center  funders  and  managers  are  becoming  increasingly 
concerned  with  controlling  productivity  and  performance  of 
operations.  For  example,  all  Federal  libraries  and 
information  centers  in  the  US.  subject  to  OMB  Circular  A- 
76  are  required  to  operate  under  a  formal  Quality 
Assurance  Surveiilance  Pian  (QASP).  This  procedure 
ensures  that  acceptable  levels  of  staff  productivity  are 
achieved  for  relevant  functions  or  activities.  In  this  chapter 
we  provide  a  brief  description  of  measures  and  methods  that 
mi^t  be  used  for  Quality  Assurance  Surveiliance. 

In  this  chapter  we  describe  measures  of  individual 
performance  of  information  center  staff.  The  performance 
measures  deal  primarily  with  services  to  users  involving 
specific  or  identifiable  center  user  services  (e.g.,  online 
database  searching,  interiibrary  borrowing,  translation, 
etc.),  operational  functions  that  involve  a  lUgh  degree  of 
professional  or  intellecnial  work  {e.g,  cataloging),  or 
production-like  activities  (e.g.,  shelving,  photocopying,  etc.). 
Below  we  list  some  functions  and  activities  for  which  we 
recommend  maintaimng  measures  of  performance  and  we 
present  examples  of  acceptable  levels  of  performance. 
Following  that  we  provide  suggested  me'hods  for  observing 
and  controlling  (in  a  quality  assurance  sense)  performance 
of  staff.  Note  that  the  process  should  be  an  evolving  one  in 
which  measures  are  observed  and  changed  to  reflect  the 
practical  aspects  of  knowledge  work.  Clearly,  performance 
ratings  must  go  beyond  the  data  collection  suggested  here,  to 
make  sure  the  measures  are  fair,  feasible,  and  practical.  The 
suggestions  below  may  go  beyond  what  an  information 
center  might  ultimately  arrive  at.  However,  they  can  serve  ts 
a  beginning  from  which  to  work. 

6.2  Performance  Levels  of  Activities  and  Functions 

The  principal  methods  of  Quality  Assurance  Surveiilance 
involve  traditional  quality  control  (QC)  procedures.  There 
are  several  aspects  of  these  procedures  that  one  must 
thoroughly  understand.  Below  we  discuss  these  aspects  and 
give  several  examples  for  each: 

•  Functions  or  activities  being  observed 

—  online  bibliographic  searching 

—  ori^nal  cataloging 

—  collection  development  and  management: 
selection  of  materials 

•  Units  or  transactions  measured 

—  fiill  searches  of  online  bioliographic  databases 

—  items  (e.g.,  books)  originally  cataloged 

—  items  (e.g.,  books,  journals,  etc.)  selected  by  the 
collection  development  process. 


•  Output  attributes  measured 

—  number  of  days  for  delivery  measured  from  a 
negotiated  delivery  date  of  full  searches  of  online 
bibliographic  databases 

—  quality  of  original  cataloging 

—  relevance  of  items  selected  by  the  collection 
development  process 

•  Acceptable  quality  levels 

—  full  searches  delivered  within  one  working  day  of 
negotiated  delivery  date 

—  items  should  be  originally  cataloged  to  center 
conformance  standards 

—  Items  selected  by  the  collection  development 
process  should  meet  center  conformance 
standards  of  relevance 

•  Refection  criteria 

—  No  more  than  5  percent  of  the  search  results 
should  be  delivered  beyond  one  day  of  negotiated 
date  of  delivery. 

—  No  more  than  10  percent  of  items  onginally 
catalogued  should  fait  to  meet  quality 
conformance  standards. 

—  No  more  than  10  percent  of  times  selected  should 
fail  to  meet  conformance  standards  of  relevance. 

•  Methods  of  surveillance 

—  Review  records  monthly  to  see  that  full  searches 
are  performed  within  acceptable  times  (i.e.,  1  day 
of  negotiated  delivery  date). 

—  Sample  20  items  originally  cataloged  every 
quarter.  T!ie  sample  catalogs  are  inspected  to 
ascertain  that  original  cataloging  conforms  to 
center  quality  standards. 

—  Sample  100  items  selected  by  the  collection 
development  process  each  year.  The  sampled 
Items  are  inspected  to  establish  that  they  meet 
center  relevance  standards. 

Note  that  the  last  two  methods  of  surveillance  above  involve 
samples.  More  detail  concerning  Quality  Control  (QC) 
sampling  is  given  in  the  next  section.  In  these  instances  the 
estimates  of  proportions  which  do  not  meet  standards  are 
subject  to  sampling  error.  Thus,  some  latitude  is  provided  in 
the  sampling  to  minimize  the  risk  of  rejecting  work  when,  in 
fact,  it  is  acceptable.  Procedures  are  discussed  in  the  next 
section  for  how  this  is  done.. 

The  QC  procedures  require  that  rejection  of  work  output  be 
based  on  a  proportion  of  units  of  output  (or  transactions) 
whicli  do  not  meet  acceptable  quality  levels.  This  method 
assumes  that  the  information  center  can  develop  meaningful 
and  relevant  conformance  standards.  Many  of  the 
information  centers  we  have  worked  with  have  developed 
such  standards.  One  can  also  set  rejection  criteria  on  other 
operational  performance  measures  as  well,  such  as 
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productivity.  Examples  of  acceptable  levels  of  productively 
might  be  two  titles  originally  cataloged  per  hour  or  eight 
invoices  processed  per  hour.  It  is  useful,  if  possible,  to  set 
productivity  levels  at  different  acceptable  levels  of  quality. 


Below  we  list  examples  of  functions  and  activities  for  which 
one  might  observe  performance  measures.  For  each  one  we 
also  provide  suggested  measures  and  the  basic  method  of 
observation. 


Collection  Development  anti  Management 

Review  and  select  materials 

1. 

Items  selected  by  the  collection  department  process  should  meet 
center  conformance  standards  of  relevance.  No  more  than  10  percent 
of  Items  selected  should  fail  to  meet  conformance  standards  of 
relevance. 

2. 

Survey  users  annually  to  determine  level  of  satisfaction  with  collection 
scope,  comprehensiveness,  and  relevance. 

3. 

Annually  review  circulation  records  and  in-house  use  to  sec  which 
items  are  never  used.  Establish  that  there  are  no  more  than  10  percent 
of  the  items  (acquired  in  the  last  two  years)  that  are  never  used. 

Review  and  select  materials  for  withdrawal 

1. 

Review  items  recommended  to  be  withdrawn  and  establish  that  fewer 
than  10  percent  are  judged  not  appropriate  for  withdrawal. 

Acquisitions 

Acquisition  of  monographs,  senals,  etc. 

1. 

Normal  orders  should  be  placed  within  five  working  days  of  request,  98 
percent  should  be  placed  within  five  working  days  and  none  over  ten 
working  days. 

2. 

Rush  orders  should  be  placed  within  48  hours  of  request. 

3. 

Normal  receipts  should  have  immediate  notification  and  receipts 
processed  within  the  hour.  All  rush  materials  should  be  processed 
within  one  day. 

4. 

Rush  receipts  should  have  immediate  notificrtion  and  receipts 
processed  within  the  hour.  All  rush  matenals  should  be  processed 
within  one  day. 

5. 

First  order  items  should  be  claimed  within  two  months  of  date  ordered 
and  monthly  thereafter.  No  claims  should  extend  beyond  two  weeks  of 
specified  time. 

6. 

Claiming  of  senals  issues  should  be  made  within  one  week  of 
identification  of  overdue  date  and  monthly  thereafter. 

7. 

95  percent  of  continuing  serials  issue  receipt  should  be  checked  in 
within  one  day  of  receipt  and  100  percent  within  two  days 

8. 

Productivity  of  ordering  and  order  control  processing  should  be  5 
Items  per  hour. 

Matenals  Receiving  and  Mail  Processing 

Materials  handling 

1. 

Incoming  materials  should  be  processed  and  delivered  within  one  day 
of  receipt.  All  items  should  be  processed  within  two  days. 

2. 

Productivity  of  mail  processing  should  be  10  normal  mail  items  per 
hour  and  4  packed  items  handled  per  hour. 

Copy  and  Enhanced  Cataloging 

1. 

Items  should  be  copy-cataloged  and  enhanced  to  information  center 
conformance  standards.  No  more  than  5  percent  of  items  should  fail  to 
meet  conformance  standards. 

2. 

Productirity  for  copy  cataloging  should  be  4  titles  per  hour. 

3. 

Throughput  time  should  be  less  than  5  days  No  more  than  5  percent  of 
titles  should  take  more  than  5  days. 

Original  Cataloging 

1. 

Items  should  be  originally  cataloged  to  information  center 
conformance  standards.  No  more  than  5  percent  of  items  should  fail  to 
meet  conformance  standards. 

2. 

Productivity  for  original  cataloging  should  be  2  titles  per  hour. 

3. 

Throughput  time  should  be  less  than  5  days.  No  more  than  5  percent  of 
titles  should  take  more  than  5  days. 

Cataloging  Maintenance 

1. 

Review  items  and  card  files  and  dispose  of  items  and/or  cards  within 
one  week  of  notification. 
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2.  Resolve  authority  conflicts  within  one  week  of  identification. 

3.  Update  automated  catalog/shelflist,  authority  file,  etc.  at  prespecified 
times. 

4.  Productivity  for  catalog  additions  should  be  2  titles  per  hour. 

Physical  Processing 

Spine  labelling,  bar  code  labelling/linking 
Other  physical  processing 
Prepare  materials  for  binding  and  repair 


I.  Productivity  should  be  8  titles  per  hour. 

2  Productivity  should  be  25  items  per  hour. 

3.  Prepare  materials  within  one  week  of  notification  by  supervisor  to  do 
so. 


Periodicals  Binding 


1.  Review  within  one  week  of  schedule. 

2.  Send  to  bindery  within  one  week  of  review 

3.  Shelve  within  one  day  of  return  lo  center. 


Invoice  Processing 

Prepare  invoices  for  payment  and  payment 
document 


1.  Prepare  within  one  week  of  receipt. 

2.  Productivity  should  be  8  invoices  processed  per  hour. 

3.  Productivity  should  be  40  invoices  per  hour. 


Online  Database  Searching 


Selective  Dissemination  of  Information 


Circulation  Activities 


Translation 


Shelving  and  Reshelving 


Interinsliiutional  Borrowing  and  Lending 
Inteiinstitutional  borrowing 


Interinstitutional  lending 


Photocopying 


1.  Full  searches  delivered  within  1  working  day  of  negotiated  date  of 
delivery.  No  more  than  5  percent  of  full  searches  should  be  delivered 
after  1  working  day  of  negotiated  date  of  delivery. 

2.  No  rush  searches  performed  beyond  negotiated  day  of  receipt. 

3.  95  percent  of  user  satisfaclio.n  ratings  at  satisfaclory  or  above  for 
relevance  of  response;  response  time;  and  number  of  items  retrieved 

1.  Searches  performed  within  two  weeks  of  scheduled  lime. 

2.  95  percent  of  user  satisfaction  ratings  at  satisfactory  or  above  for 
relevance  of  response;  response  time;  and  number  of  items  reinevcd 

1.  Surveys  should  establish  that  fewer  than  5  percent  of  the  users  arc 
dissatisfied  or  very  dissatisfied  with  speed  of  response  al  circulation 

2.  Random  spot  checks  should  establish  that  check-in  and  check-out  are 
done  expeditiously  and  unnecessary  queues  do  not  form 

1.  No  more  than  2  percent  of  translations  completed  more  than  10  days 
beyond  negotiated  time. 

2.  No  more  than  1  percent  of  rush  requests  within  two  days  beyond 
negotiated  lime. 

3.  No  more  than  2  percent  of  user  ratings  less  than  satisfied  or  very 
satisfied  for  response  time,  accuracy  of  translation,  etc 

1.  Productivity  of  shelving  should  be:  125—150  volumes  per  hour  (bound 
journal  volumes),  150—175  issues  per  hour  (loose  unbound  issues). 
Sorting  productivity  should  be  5  to  10  minutes  per  truck  (depending 
upon  size  of  truck). 

2.  No  more  than  1  percent  of  ILL  or  circulated  items  shelved  incorrectly. 


1.  No  more  than  10  percent  of  normal  borrowing  requests  should  be 
processed  in  more  than  1  day. 

2  All  rush  requests  should  be  processed  within  48  hours. 

3.  Productivity  of  requests  fulfilled  should  be  50  requests  pei  hour  (not 
including  photocopying). 

1.  No  more  than  10  percent  of  lending  requests  processed  in  more  than  3 
days. 

2.  Productivity  should  be  25  initial  requests  processed  per  hour  and  4 
lending  items  fulfilled  jici  hour. 

Productivity  of  photocopying  should  be  30  articles  per  hour. 

2.  No  more  than  2  percent  of  user  ratings  less  than  satisfied  or  very 
satisfied  for  response  time  and  quality  of  photocopy. 


53 


6.3  Data  Collection  Methods  for  Quality  Assurance 
Surveillance 

Data  collection  for  staff  performance  evaluation  is 
conducted  to  help  ensure  that  sufficient  levels  of 
performance  are  achieved.  To  do  this  we  suggest  that  several 
measures  be  observed  including:  productivity,  timclinc.ss  of 
services  and  quality  of  some  activities  and  services.  There 
are  essentially  six  methods  employed  in  rating  staff 
performance,  as  follows: 

1.  Observation  of  Productivity.  Productivity  is  measured 
by  the  number  of  items  produced  per  work  hour. 
Output  quantities  may  be  observed  by  logs  or  other 
means  over  a  specified  period  of  lime  (e.g.,  one  month 
or  quarter)  and  corresponding  input  hours  observed  by 
logs,  diaries,  random  observations,  etc. 

2.  100%  Inspection  or  Observation  of  the  Timeliness  with 
Which  Activities  Are  Performed  or  Services  Provided. 
Timeliness  is  determined  by  recording  when  events 
take  place.  Standards  for  the  timeliness  of  an  activity 
are  based  on  a  proportion  of  events  that  arc  to  be 
accomplished  within  a  specified  time  period.  For 
example,  acceptable  performance  might  be  that  no 
events  be  accomplished  beyond  specified  time  or  fewer 
than  3  percent  accomplished  beyond  specified  lime  or 
fewer  than  3  percent  accomplished  beyond  that 
specified  time.  All  events  (100%)  would  be  observed  to 
make  sure  that  acceptable  levels  of  standards  of 
timeliness  arc  achieved. 

3.  Random  Sampling  of  Output.  In  most  instances, 
individual  observations  of  performance  arc  difficult  or 
expensive  In  these  instances,  the  output  of  activities  is 
sampled  and  inspected  for  timeliness  or  quality. 

4.  Random  Spot  Checks.  Spot  checks  ascertain  that  some 
activities  arc  being  performed  as  specified.  For 
example,  the  circulation  desk  might  be  observed  at 
random  times  to  insure  that  it  is  manned,  large  queues 
have  not  formed,  and  users  are  being  served  properly, 
etc.  A  spot  check  performed  weekly  or  monthly  can 
verify  that  this  is  happening. 

5.  User  Feedback  Including  User  Siuveys.  A  survey  of 
users  may  be  done  in  order  to  ascertain  that 
performance  standards  arc  being  met  (e.g.,  that  a 
standard  for  reference  timeliness  and  quality  is 
maintained  from  the  user  perspective).  See  Part  3  for 
in-depth  discussions  of  user  surveys. 

6.  Validated  User  Complaints.  User  complaints  should  be 
encouraged.  They  should  be  received  and  verified  as  a 
means  of  assuring  quality  of  work  performed.  All 
complaints  should  be  responded  to  and  the  reason  for 
the  problems  determined  and  addressed. 

Each  of  these  methods  is  described  in  more  detail  below. 

Performance  of  a  required  operational  function  or  service  is 
considered  acceptable  when  the  number  of  discrepancies 
such  as  rush  requests  filled  in  more  than  48  hours  is  below  a 
predetermined  proportion  of  transactions.  Most  levels  of 
acceptance  found  through  100  percent  inspection  are  stated 
in  terms  of  percentages.  The  percentages  should  be 
computed  periodically  (usually  monthly)  by  dividing  the 
number  of  discrepancies  by  a  lot  size  or  sample  frame 
(usually  monthly  or  quarterly  output).  For  required  services 


inspected  by  random  sampling,  the  number  of  faulty  items 
or  services  which  determine  a  lot  rejection  is  determined  by 
an  acceptance  level  and  the  statistical  quality  control 
methods  applied. 

In  many  instances,  the  acceptance  or  rejection  of 
performance  will  be  detennined  by  10'  %  inspection  made 
on  a  periodic  basis  (usually  monthly)  In  order  to  assist 
100%  inspection  to  observe  timelessness,  it  will  be 
necessary  to  maintain  records  (logs)  of  transactions  so  that 
timeliness  of  events  can  be  determined.  In  the  example 
above,  a  log  must  be  maintained  for  each  order  covenng  the 
date  of  request,  date  of  order  placement,  date  of  return,  date 
of  delivery,  and  so  on.  The  logs  (maintained  manually  or  on 
an  automated  .system)  could  highliglit  activities  that  exceed 
performance  standards  of  timeliness.  In  other  instances, 
timeliness  of  activities,  such  as  delivery  of  periodic  reports, 
can  be  observed  through  review  of  each  report  and  its 
delivery. 

In  order  to  assess  timeliness,  a  set  of  guides  are  provided  at 
the  end  of  this  chapter.  These  guides  (Observation  of 
Timeliness  Guides  Numbers  1  through  3)  are  based  on 
transactions  which  require  records  (logs)  under  any 
circumstances  and  thus,  records  of  timeliness  (i.e.,  time  at 
which  events  are  performed)  are  merely  additions  to  the 
existing  logs.  For  example,  a  record  is  usually  maintained 
when  a  request  is  made  for  an  item  (see  Observation  of 
Timeliness  Guide  No.  1 ).  Examples  of  events  which  must  be 
recorded  for  that  item  to  ascertain  timeliness  are:  dales  of 
original  request,  when  order  is  placed,  claiming,  receipt  of 
item  processed,  check  in  circulated  item,  modify  copy,  or 
original  cataloging.  Obviously,  these  events  involve  several 
functions  including  acquisition  and  cataloging.  In  order  to 
inspect  timeliness  standards  for  special  cases,  it  is  necessary 
to  identify  special  classes  of  transactions.  In  the  example 
above,  this  would  include  distinguishing  between  normal 
and  rush  orders;  new  serials  or  continued  serials,  and  so  on. 

The  surveillance  records  can  be  maintained  in  any  one  of 
several  ways  including  logs  or  computer  records.  Such  logs 
or  records  should  have  some  mechanism(s)  for  highlighti.ig 
events  that  fall  below  the  acceptance  standards.  Otherwise, 
the  logs  or  records  must  be  perused  monthly  to  identify 
instances  where  the  standards  are  exceeded. 

The  second  principal  method  of  inspection  is  random 
sampling.  In  this  case,  a  periodic  (usually  quarterly)  lot  of 
output  is  sampled  and  inspected.  For  example,  the 
acceptance  level  or  original  cataloging  is  that  no  more  than 
five  percent  of  the  items  onginally  cataloged  shall  fail  to 
conform  to  information  center  standards.  A  supervisor,  or 
someone  designated  by  the  supervisor,  should  determine 
whether  sample  outputs  conform  to  information  center 
standards.  For  example,  there  might  be  600  items  originally 
cataloged  annually  at  the  information  center  standards. 
These  are  subdivided  into  quarterly  output  and  randomly 
sampled  using  the  following  methods.  There  are  about  150 
Items  onginally  cataloged  quarterly.  These  are  the  lots  to  be 
sampled.  There  is  a  standard  statistical  guide  for 
determining  sample  size  based  on  the  lot  size.  This  guide 
recommends  a  normal  sample  size  of  20  for  a  lot  size  of  150 
(see  Table  6.1).  The  level  of  acceptance  and  rejection  is  given 
in  Table  6.2.  The  acceptance  levels  in  the  table  include  4.0 
percent  and  6.5  percent,  but  not  5  percent,  thus  6.5  (higher) 
is  used. 
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TABLE  6.1 

SAMPLE  SIZES  TO  USE  AS  A  FUNCTION 
OF  LOT  SIZE  AND  PRECISION  REQUIRED 


Lot  Sizo 

Normal 

2-6 

2 

9  O', 

3 

16*25 

S 

26-50 

6 

SI  -90 

13 

91  •  150 

20 

15!  •  260 

32 

261  -600 

50 

501  - 1.200 

80 

1  201  -  3,200 

125 

3.201  -10,000 

200 

10.00!  -  35,000 

315 

35.00!  •  150.000 

500 

150.001  -500.000 

800 

500.0C0  and  ovar 

1250 
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TABLE  6.2 

MIL-STD  105D  ACCEPTANCE,  REJECTION  LEVELS  FOR 
NORMAL  INSPECTION 

Acceptabto  quaBty  Lovals* 

40% 

6.5% 

10-0%- 

Size 

AC 

Re 

Ac.  Re 

Ac 

Ro 

8» 

+ 

+ 

1  2 

2 

3 

13 

1 

2 

2  3 

3 

4 

20 

2 

3 

3  4 

5 

6 

~Typ4oal  taoies  have  sois  of  1 5  sample  sires  and  26  Acceptable  Quabty  Leva's 


The  table  entry  for  6.5  peicent  and  samplesizeof  20is3  and 
4,  which  means  that  nonconformance  of  3  or  fewer 
cataloged  items  from  the  sample  of  20  from  a  quarterly  lot  of 
approximately  125  will  constitute  acceptance  of  that  lot  and 
nonconformance  of  4  or  more  cataloged  items  from  the 
sample  of  20  means  that  the  lot  does  not  meet  an  acceptable 
level  of  performance.  It  is  noted  that  the  rejection  level  of  3 
nonconformed  items  corresponds  to  IS  percent  i.e.,  3 
divided  by  20)  which  permits  10  percent  (15  minus  5%) 
leeway  due  to  sampling  error.  The  items  to  be  randomly 
sampled  would  be  determined  by  the  supervisor  or  other 
designated  person. 

There  are  basically  two  ways  in  which  random  samples 
should  be  selected.  The  first  of  these  involves  the  use  of 
random  tables.  A  table  of  random  numbers  is  given  below.  In 
the  first  sampling  procedure,  a  lot  from  which  the  sample 
will  be  drawn  is  listed  or  ordered  so  that  one  can  identify  an 
item  with  a  number  (from  1  to  the  number  of  items  in  the 


lot).  Then  random  numbers  are  chosen  from  the  table  and 
the  numbers  assigned  to  those  designated  in  the  lot.  This 
represents  the  sample.  The  second  method  involves 
systematic  random  sampling.  In  this  procedure,  the  lot  size 
(i.e.,  number  of  items  in  the  lot)  is  divided  by  the  sample  size 
to  determine  a  sampling  interval  (e.g.,  every  tenth  or  thirty- 
second  record).  Then  the  first  sample  item  is  chosen  by 
choosing  a  random  number  from  one  to  the  number  in  the 
sample  interval  (e.g.,  10  or  32)  using  the  table  of  random 
numbers.  The  remaining  sampled  items  are  chosen 
systematically  by  counting  the  interval  (10  or  32).  Thus,  if 
the  sampling  interval  is  32  and  the  first  random  number  is 
24,thesamplesare24, 56, 88, 120, 152,  etc.  Instructions  on 
how  to  use  the  random  table  are  given  below. 

The  random  numbers  in  the  table  are  arranged  in  groups  of 
five  numbers  (i.e.,  5G651,  25480,  etc.).  To  use  the  table, 
begin  by  picking  at  random  a  group  of  numbers  on  any  page 
of  the  table.  Various  patterns  should  be  used  alternately.  For 
one  sample  one  can  use  rows,  for  the  next  sample  one  can 
use  columns,  and  for  the  third  sample  one  can  establish  a 
diagonal  pattern. 


TABLE  6.3 

EXAMPLE  OF  RANDOM  NUMBER  TABLE  PAGE 


Rows* 

1 

Columns* 

2 

3 

1 

58651 

25480 

48689 

2 

90578 

71705 

43472 

3 

31535 

2172S 

32554 

4 

4T490 

16907 

68179 

5 

48159 

39410 

05665 

•  Nol©  that  typical  fandom  taW©$  hava  a$  many  as  750  fiv©  diQit  sals  of  random 
numbarf  par  paga 


The  use  of  variety  in  the  Random  Number  Table  ensures 
that  patterns  that  might  be  detected  by  staff  do  not  occur. 
Besides  starting  at  different  random  points  and  alternating 
thp  patterns  for  finding  a  string  of  random  numbers,  die  user 
may  at  some  point  in  time  wish  to  use  the  first  significant 
digits  instead  of  the  last.  For  instance,  in  the  random  number 
group  58651  the  last  three  digits  (i.c.,  651)  arc  used  when 
looking  for  a  random  number  with  three  digits,  but  there  is 
no  reason  why  one  could  not,  for  a  period  of  time,  use  the 
first  three  digits  (i.e.,  586). 

The  person  responsible  for  quality  assurance  should  first 
select  the  areas  and  times  for  random  sampling  using  the 
procedures  in  the  sampling  guides  and  program  these 
inspections  on  the  schedule.  When  the  sampling  procedure 
does  not  allow  for  specific  selection  of  inspection  items 
during  the  preceding  quarter,  it  would  be  necessary  to  show 
on  the  schedule  the  date  and  time  the  sample  selection  will 
be  accomplished  and  when  the  inspex-rinns  wi!!  be 
conducted.  Then  one  should  program  into  the  schedule  the 
penodic  checklist  items  to  be  reviewed  during  the  period. 
This  schedule  should  show  what  the  responsible  person  is 
monitoring  each  day. 

Tally  checklists  should  be  prepared  for  each  sampling  guide 
and  used  to  record  information  on  observations  and  defects. 
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Each  observation  in  the  sampling  should  be  recorded  on  the 
tally  checklists.  These  documents  then  become  formal 
record  for  later  reference.  The  tally  of  observations  and 
defects  at  the  end  of  the  month  would  be  compared  to  the 
acceptable  number  of  defects  appearing  in  the  appropriate 
sampling  guide.  Any  errors  detected  during  the  course  of  the 
surveillance,  even  if  not  of  sufficient  degree  to  render  the 
service  unsatisifactory  in  terms  of  the  acceptance  level 
parameters,  would  require  corrective  action  by  the 
supervisor,  if  possible. 

In  some  instances,  inspection  requires  visual  observation  of 
activities  or  events.  In  order  to  achieve  such  inspection, 
random  spot  checks  of  the  operation  could  be  made  by  a 
supervisor  or  manager.  Such  spot  checks  should  be  made  on 
a  weekly  and  monthly  basis  depending  on  the  observation 
made.  Note  that  the  process  of  spot  checks  will  have  the 
effect  of  the  staff  knowing  certain  activities  have  to  be  done 
and  on  time. 

In  many  instances,  it  is  necessary  to  inspect  a  report  or 
output  of  an  activity  to  determine  whether  it  is  provided 
within  quality  standards.  Examples  of  functions  and 
activities  for  these  arc  listed  below. 

1)  Monthly  review  of  records  to  see  that  titles  and  items 
requested  are  the  same  as  titles  received  unless  not  tn 
print. 

2)  Monthly  review  of  records  to  see  that  review  for 
subscription  claiming  is  done  twice  per  year. 

3)  Special  reports  and  other  materials  meet  standards  set 
for  content  and  format  and  time  specified. 

4)  Observe  repairs  to  see  that  repair  process  meets 
specifications  for  all  items  involved. 

5)  Observe  inspection  reports  to  sec  that  security 
inspection  includes  all  structural  and  environmental 
features,  the  log  meets  specifications  and  performance 
of  safety  officers  meets  requirements. 

6)  Review  and  compare  all  items  chosen  and  deleted  by 
selection  staff  to  see  that  they  are  appropriate. 

7)  Monthly  review  of  records  to  determine  if  journal 
continuity  is  maintained. 

8)  Meet  with  staff  to  determine  that  they  maintain 
currency  of  knowledge  through  specified  means  and 
events  and  that  this  is  reported  in  monthly  reports. 

9)  Review  records  to  determine  that  interlibrary 
borrowing  does  not  involve  more  than  5  requests  for  a 
title. 

10)  Monthly  review  of  records  to  determine  that  incoming 
and  outgoing  mail  is  delivered  correctly  and  messenger 
assignments  are  accomplished  successfully. 

OBSERVATION  OF  TIMELINESS 
GUIDE  NO.  1 

1.  Function(s) 

Acquire  materials  fui  the  Iiifuinidtiuii  Center. 
Catalog  and  Classify  Materials  for  the  Information 
Center. 

2.  Transactions 

A  transaction  is  acquisition  of  specific  items  (i.c., 
monographs,  serials,  etc.).  Events  associated  with  each 


transaction  are  traced  from  request  through  technical 
processing. 

3.  Events 

Events  in  which  time  should  be  recorded  include: 
Request 
Order  placed 
Receipt  processed 
In-house  routing  of  serial  issues 
Claiming  of  1st  order  items 
Claiming  of  serial  issues 
Modify  OCLC  copy 
Original  cataloging 

4.  Special  Classlftcalions  (Attributes) 

Materials  need  to  be  identified  in  the  following  ways: 
Normal  orders 
Rush  orders 
New  serials 
Continued  serials 
Continued  serials  (without  lapse) 

5.  Surveillance  Method 

The  surveillance  method  involves  inspection  of  all 
acquired  items,  the  events  listed  above  and  when  they 
took  place. 

OBSERVATION  OF  TIMELINESS 
GUIDE  NO.  2 

1.  Funct!on(s) 

Provide  Information,  Reference,  and  Research 
Assistance  Services 

2.  Transactions 

Transactions  arc  reference  questions  or  full  searches 
initiated  by  a  user 

3.  Events 
Request 

Interview  with  user 
Searching,  packaging,  and  dispatch 

4.  Special  Classifications  (Attributes) 

Rush  searches 
Walk-in  requests 
Telephone  requests 

5.  Surveillance  Method 

Review  records  monthly  to  see  that  events  are 
performed  within  standard  times. 

OBSERVATION  OF  TIMELINESS 
GUIDE  NO.  3 

1.  Funclion(s) 

Piovide  Ciieulalioii  Service  for  and  Physical  Access  to 
Information  Center  Materials 

2.  Transactions 

Transactions  are  circulated  items;  interlibrary 
borrowing  requests;  interlibrary  loan  requests;  and 
access  through  document  delivery  services. 
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3.  Events 

Item  need  determined 
Response 

Follow-up  overdues 

ILB  or  document  delivery  vendor  request 

Repeat  ILB  or  document  delivery  vendor  request 

ILB  or  document  delivery  vendor  item  sent  to  user 

ILL  request 

ILL  discharge 

Forms  filed 

Materials  shelved 

4.  Special  Classifications  (Attributes) 

Rush  requests 

5.  Surveillance  Method 

Review  request  forms  to  see  that  events  arc  performed 
within  standard  times. 


RANDOM  SAMPLING 
GUIDE  NO.  1 

1.  Function(s) 

Catalog  and  Classify  Materials 

2.  Unit  of  Observation 
Items  originally  cataloged 

3.  Acceptable  Quality  Level(s) 

Over  5.0  percent  of  items  arc  not  cataloged  to 
information  center  conformance  standards 

4.  Lot  Size  for  Sampling 

All  quarterly  items  cataloged 

5.  Sampling  Size 

6.  Sampling  Procedures 

Divide  the  lot  size  by  20  which  determines  the 
sampling  intervals  (c.g ,  every  6th  record).  Then  choose 
the  first  sampled  item  by  choosing  a  random  number 
from  one  to  the  number  in  the  sample  internal  (i.e.,  6). 
The  remaining  sampled  items  will  be  chosen 
systemically  by  counting  the  interval  (6). 


7.  Inspection  Procedures 

The  items  cataloged  will  be  inspected  to  ascertain  that 
original  cataloging  conforms  to  standards. 

8.  Performance  Criteria 

Performance  is  satisfactory  when  three  or  fewer  items 
are  not  originally  cataloged  to  conformance  standards 
and  performance  is  unsatisfactory  if  four  or  more  items 
arc  not  cataloged  to  conformance  standards. 

Random  SAMPLING 
GUIDE  NO.  2 

1.  Function(s) 

Provide  Information,  Reference,  and  Research 
Assistance  services. 

2.  Unit  of  Observation 

(a)  A  completed  online  search 

(b)  A  completed  manual  search 

3.  Acceptable  Quantity  Level(s) 

Over  6.5  percent  of  database  searched  online  not 
relevant  to  question. 

4.  Lot  Size  of  Sampling 

All  on  line  searches  performed  in  a  quarter. 

5.  Sampling  Size 

50  online  searches  quarterly. 

6.  Sampling  Procedures 

A  systematic  random  sampling  of  online  searches 
where  the  sampling  starting  point  is  vaned  weekly 

7  Inspection  Procedures 

Each  week  the  online  searches  are  identified  and  the 
search  procedures  and  output  arc  reviewed  from  the 
search  records. 

8.  Performance  Criteria 

Performance  is  satisfactory  if  seven  or  fewer  databases 
are  not  relevant  or  do  not  show  relevant  items  and 
unsatisfactory  performance  is  when  eight  or  more 
searches  do  not  meet  these  standards. 


Part  3 


Evaluation  of  Information 
Center  Effectiveness 
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Chapter  7 

Statistical  Survey  Methods  for  Evaluating 
Information  Center  Effectiveness 


7.1  Background 

In  this  chapter  we  discuss  statistical  surs’ey  methods  used  for 
evaluating  information  center  effectiveness.  In  particular, 
these  statistical  surveys  involve  the  middle  perspective  on 
the  framework  given  in  Figure  4  in  Chapter  2.  In 'these 
surveys  are  such  user-related  measures  as  user  input  costs 
(E  on  the  figure),  output  quantities  and  attributes  (F),  and 
activities  and  resources  usi^  in  work  (D).  More  specifically, 
these  surveys  focus  on  information  resources  used  (D)  and 
the  interface  (f)  with  information  center  output  quality, 
timeliness,  accessibility  and  availability.  In  other  chapters  of 
this  Part  on  Evaluation  of  Information  Center  Effectiveness, 
we  give  specific  examples  of  survey  methods  applied  to 
evaluatii'g  the  use  and  usefulness  of  information  centers 
(Chapter  8),  and  use  and  usefulness  of  specific  services 
(Chapter  9),  as  well  as  higher  order  effects  of  information 
qse  and  the  contributions  that  information  centers  make  to 
information  use  (Chapter  10).  In  this  chapter  we  focus  on 
some  fundamentals  of  statistical  survey  methods  that  can  be 
used  for  any  of  the  statistical  surveys  applied  in  evaluating 
Laformation  centers. 

There  are  many  instances  in  evaluation  in  which  data  about 
information  center  services  need  to  be  collected  from 
individuals.  Such  data  may  provide  infonnation  about  who 
uses  and  does  not  use  center  services  and,  more  importantly, 
why  services  are  not  used.  Such  information  can  also  be  used 
to  establish  the  use,  usefulness  and  value  of  information 
services  on  users’  work.  Thus,  one  can  link  the  performance 
of  services  in  terms  of  their  quality,  timeliness,  availability, 
accessibility,  etc.  to  their  purpose  of  use,  and  how  their  use 
affects  user’s  work.  A  substantial  number  of  evaluation 
studies  involve  users  of  information  services.  In  this  chapter 
we  emphasize  examples  of  survey  methods  that  might  be 
employed  in  such  evaluation  studies.  Again,  we  focus  largely 
on  survey  examples  that  fit  the  overall  evaluation  framework 
discussed  in  Part  1,  Chapter  2  of  this  Manual.  Note  that  the 
methods  described  here  apply  equally  well  to  other  survey 
measures  and  evaluation  studies. 

User  studies  cited  in  the  literature,  for  the  most  part, 
investigate  users  of  specific  information  services,  products 
or  systems.  The  survey  methods  described  in  subsequent 
chapters  provide  a  comprehensive  set  of  surveys  that, 
together,  provide  a  "complete”  picture  of  the  entire  range  of 
services  provided  by  information  centers.  Furthermore,  the 
surveys  carefully  tie  operational  performance  (i.e., 
relationships  of  a  center’s  input  costs  of  resources  and  their 
attributes  and  output  quantities  and  attributes)  to  service 
effectivenm.  The  focus  of  evaluation  surveys  is  on  users  and 
prospective  uscre.  '.Vc  try  to  obtain  data  that  indicate  why 
services  are  and  are  not  used,  and  what  the  consequences  of 
use  and  non-use,  are- to  the  information  center  and  the 
center’s  and/or  user’s  parent  organization. 

The  investment  an  information  center  makes  in  both  time 
and  staff  resoure’es  to  conduct  evaluation  suiveys  will  vary 


with  each  center  and. the  services  being  evaluated.  In  every 
case,  however,  management  should  ask  what  results  will  be 
achieved  from  the  survey  before  it  is  designed  and 
implemented.  Evaluation  surveys,  in  themselves,  will  not 
result  in  change.  The  survey  results  are  merely  instruments 
for  ehange,  to  be  used  with  other  sources  of  information  and 
knowledge  to  be  applied  with  good  judgement.  We  strongly 
believe  that  evaluation  surveys  should  be  done  only  if 
decisions  are  going  to  be  based  on  them.  We  also 
recommend  that  such  decisions  should  be  made  within  the 
framework  of  the  information  center’s  mission,  goals  and 
objectives  (sec  Chapter  1).  With  measurable  center 
objectives,  one  can  derive  meaningful  evaluation  goals  and 
measurable  center  objectives.  For  example,  a  center 
objective  for  a  particular  service  might  be  to  provide  X 
number  of  transactions,  of  which  80  percent  are  to  be  rated 
for  quality  and  timeliness  as  satisfactory  or  very  satisfactory 
by  users.  The  evaluation  survey  can  then  be  designed  to 
measure  the  number  of  transactions  and  ratings  of 
satisfaction  with  quality  and  timeliness.  In  addition,  the 
survey  can  be  designed  to  identify  factors  that  affect  the 
number  of  transactions  and  user  satisfaction,  so  that 
decisions  can  be  made  to  help  increase  the  number  of 
transactions  or  satisfaction  ratings,  if  they  are  found  to  be 
too  low.  For  example,  satisfaction  with  timeliness  of  service 
provision  might  be  that  some  transactions  require  fast 
response  and  others  don’t.  Thus,  a  decision  can  be  made  to 
handle  ’’rush”  requests  differently  from  others.  Or  it  might 
be  that  the  distance  of  some  users  from  the  information 
center  requires  different  response  mechanisms.  Number  of 
transactions  can  depend  on  many  factors,  such  as  inherent 
need  for  the  service,  competition  from  other  sources, 
awareness  of  the  service  and  service  performance.  Thus, 
data  can  be  gathered  about  these  factors  to  aid  in  decisions 
relating  to  the  number  of  transactions  stated  m  the 
measurable  objective. 

Evaluation  surveys  provide  merely  a  snapshot  in  time.  To  be 
an  effective  management  tool,  one  must  consider  the 
possibility  of  making  surveys  part  of  an  ongoing  process  to 
provide  relevant  management  information.  Surveys  do  not 
normaily  need  to  be  conducted  weekly  or  monthly. 
However,  one  should  adapt  procedures  and  a  system  of 
measures  about  users  that  will  yield  data  at  appropriate 
intervals  of  time.  For  example,  if  a  center  objective  is  to 
increase  levels  of  transactions  (from  X  number  to  Y 
number)  and  satisfaction  ratings  (from  60%  satisfied  to  80% 
satisfied)  over  three  yea.'s,  one  must  be  able  to  measure 
progress  each  year  and  to  determine  the  effects  of  decisions 
(e.g.,  to  handle  “rush”  requests  differently,  to  increase 
awareness  of  the  service,  etc.). 

Finally,  by  knowing  and  understanding  operational 
performance  (i,e.,  cost  per  transaction  at  various  levels  of 
quality  and  timeliness),  center  management  can  estimate 
what  it  would  cost  to  achieve  the  center  objective  of 
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increased  transactions  and  satisfaction  ratings. 
Furthermore,  one  can  speculate  with  some  degree  of 
confidence  as  to  what  effect  achieving  'he  objective  will  have 
on  users*  work.  This  presents  a  much  stronger  case  to 
funders  for  increasing  budgets.  Thus,  evaluation  surveys  and 
operational  evaluation  together.  <»n  facilitate  planning, 
management  and  administratidh,  which  can  lead  to  better 
service.  This  requires  obtaining  reliable  and  useful  data  and 
information  from  evaluation  surveys.  Below  we  describe 
survey  methods  that  should  achieve  this  objective. 

7.2  Components  of  Survey  Design  and  Application 

There  are  eight  basic  components  of  survey  design  and 
application  as  follows; 

•  Overall  evaluation  design 

•  Data  collection  methods 

•  Sample  design 

•  Questioimaire  design 

•  Data  processing 

•  Data  analysis 

•  Statistical  analysis 

•  Presentation. 

Surveys  are  like  a  chain  in  which  each  process  in  a  survey, 
like  links  in  a  chain,  are  equaliy  important.  If  one  proce.ss  is 
done  improperly,  the  entire  survey  results  can  be  mined  or 
invalid  results  obtained,  regardless  of  how  well  all  the  other 
processes  arc  done.  The  eight  components  above  arc 
discussed  in  this  and  subsequent  chapters. 

7.3  Overall  Evaluatiua  Design 

Tlie  methods  employed  in  surveys  depend  on  the  overall 
evaluation  design  which  is  dependent  on; 

•  the  objectives  of  the  evaluation  and  the  decisions  that 
will  be  made  as  a  result  of  the  evaluation; 

•  who  or  what  will  be  affected  by  the  decisions  (i.c., 
center  staff,  management,  users,  hinders,  etc.); 

•  the  overall  environment  in  which  the  information 
center  operates  inciuding  such  factors  as  a  single 
versus  multiple  sites  served  by  the  center,  number  in 
the  population  served,  whether  or  not  the  population 
served  is  in  (i.e,  employed  by)  the  same  organization  as 
the  center;  and 

•  the  current  knowledge  concerning  center  users  and 
prospective  users. 

These  issues  dictate  to  a  large  degree  the  survey  methods 
used  for  data  collection,  sample  design,  questionnaire 
design,  and  the  other  components  listed  above.  We 
emphasize  that  there  is  also  an  interdependence  among  the 
components.  For  exampie,  data  collection  methods  (e.g., 
self-administered  questionnaire,  telephone  interview  or 
personal  interview)  will  dictate,  in  part,  the  questionnaire 
design.  How  the  questionnaire  is  designed  will  affect  data 
processing,  and  so  on. 

7.4  Survey  Data  Collection  Methods 

Surveys  obviously  require  input  from  respondents  drawn 
from  the  population  being  surveyed.  Surveys  should  be 
thought  of  as  part  of  a  communication  process,  where  the 
researcher  (evaluator)  is  communicating  with  the 
respondent  (data  provider).  This  communication  is  a  two- 
way  process  in  that  the  wording  of  questions  is  the 


researcher's  method  of  communicating  and  responses  to  the 
questions  form  the  respondents'  communication. 
Sometimes  we  ask  respondents  if  they  would  like  to  sec  the 
survey  results.  The  communication  process  .an  be 
characterized  by  the  degree  of  interaction  between  the 
respondent  and  the  evaluator.  The  utility  of  various  methods 
depends  on  such  factors  as  required  accuracy  of 
communication,  time  and  cost  constraints.  There  are  six 
basic  survey  data  collection  methods  that  might  be  used  for 
evaluation  purposes^’ 

•  Self-aclminisiered  questionnaires.  This  is  probably  the 
most  commonly  used  data  collection  method.  The 
method  involves  (1)  sending  the  questionnaire  to 
respondents  through  the  mail  or  the  organization's 
distribution  or  (2)  handing  the  questionnaire  to  an 
information  center  user,  who  is  asked  to  fill  it  out  and 
return  it  to  the  center  or  evaluator.  This  method  is 
inexpensive  and  has  the  advantage  of  letting 
respondents  fill  out  the  questionnaires  at  their  own 
pace  and  to  think  about  their  responses  more 
thoroughly.  However,  this  method  has  the  distinct 
disadvantage  of  potential  low  response  rates.  Typical 
mailed  questionnaires,  for  example,  often  result  in  less 
than  20  percent  responses  and  questionnaires  handed 
out  in  information  centers  often  result  in  less  than  one- 
third  responses.  Those  who  voluntarily  complete  a 
questionnaire  may  be  different  from  whose  who  do  not 
(e.g.,  they  may  be  frequent  users,  less  busy,  better 
educated,  etc.).  Generally,  one  should  aim  for  more 
than  50  percent  response  rates.  From  a  cost 
standpoint,  it  is  far  belter  to  have  a  smaller  initial 
sample,  and  achieve  higlier  response  rates  (through 
telephone  follow-ups  to  non-respondents  or  some 
other  means  of  follow-up)  than  to  have  a  large  sample 
with  low  response  rates.  If  all  the  information  that  is 
collected  is  for  measures  and  derived  indicators,  a 
combination  of  telephone  interviews  and  self- 
administered  questionnaires  will  suffice.  For  example, 
when  visitors  leave  the  center,  they  can  be  asked  to  fill 
out  questionnaires  at  that  time  (with  tables  or  desks  set 
aside  to  do  this)  or  asked  to  fill  them  out  at  their  offices 
and  return  them  (on  subsequent  visits  or  by  mail).  The 
sampled  individual  can  be  asked  for  name  and 
telephone  number  to  permit  telephone  follow-up  (to 
clarify  responses  if  necessary)  or  to  obtain  the 
information  over  the  telephone  if  the  person  has  not 
responded.  If  the  information  center  is  part  of  the 
respondents'  parent  organization,  one  can  expect 
higher  response  rales  than  if  it  is  not.  We  expect  about 
40—50  percent  initial  response  rates  for  professionals 
employed  by  the  center's  parent  organization  and 
sufficient  responses  are  achieved  by  letter  follow-up 
and  then  telephone  follow-up,  if  necessary,  to  prompt 
response  or  to  actually  collect  data.  Only  about  one- 
fifth  of  the  surveys  we  have  performed  in  this 
environment  have  required  telephone  follow-up. 
Another  aspect  of  self-administered  questionnaires  is 
that  the  questiormaire  must  be  earefully  structured  and 
worded  with  sutheient  instructions  and  definitions  of 
terms. 

•  Observation  of  users  in  the  information  center.  Another 
method  is  to  merely  observe  users  in  the  eenter  to 
determine  what  services  and  resources  they  use.  If  the 
observations  can  be  made  unobtrusively,  one  can 
measure  extent  of  use  of  theserviees  and  resources.  On 
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the  other  handj  the  method  is  very  limited  in  the 
number  of  measures  that  can  be  obtained  and  must  be 
accompanied  by  collecting  data  (from  the  same 
persons)  on  frequency  of  visits  and  use,  satisfaction, 
and  so  on. 

•  Telephone  inietviews.  TTie  telephone  interview  is  an 
excellent  method ,  for  collecting  survey  data.  This 
method  should  achieve  a  higher  response  rate  than 
self-administered  questionnaires,  but  less  than 
personal  interviews  (see  below).  The  time  taken  to 
complete  the  survey  am  also,  be  lower  than  for  self- 
administered  questionnaires  so,  this  method  is  often 
used  if  the  evaluation  has  to  be  completed  in  a  short 
timehame.  The  telephone  interview  questionnaire  can 
be  less  structured  than  the  self-administered 
questionnaire  and  communication  is  enhanced  by 
letting  respondents  ask  questions  and  interviewers 
clarify  responses.  There  are  some  drawbacks  to 
telephone  interviews: 

—  They  are  more  expensive  than  self-administered 
questionnaires,  but  less  expensive  than  personal 
interviews. 

—  Interviewers  must  be  trained. 

—  They  require  staff,  telephones  and  facilities  to  do 
the  interviewing. 

—  Ihere  must  be  a  current  and  complete  telephone 
directory,  although  this  should  not  be  a  problem 
when  respondents  are  employed  in  the  center's 
parent  organization. 

•  Personal  interviews.  Personal  interviews  are  conducted 
by  having  interviewers  ask  questions  from  a  survey 
instrument  or  interview  guide.  This  method  can  be 
used  only  when  it  is  possible  to  communicate  with 
respondents  directly  such  as  in  the  information  center 
or  at  the  respondents’  place  of  work.  The  interviewers 
normally  ask  specific  questions  and  record  the 
answers.  They  can  also  clarify  the  intent  or  meaning  of 
questions,  if  asked.  Most  of  the  data  obtained  for 
effectiveness  measures  can  be  precoded  and,  therefore, 
recording  bias  can  be  kept  to  a  minimum.  Personal 
interviews  have  the  advantage  of  permitting  complex 
questions  and  lengthy  interviews.  However,  neither  of 
these  two  conditions  are  necessary  for  data  collection 
for  effectiveness  measures.  On  the  other  hand,  the  cost 
of  personal  interviews  is  high  because  interviewers 
should  be  trained  and,  if  the  interviews  are  conducted 
in  workplaces,  there  are, substantial  additional  costs 
associated  with  setting  up  the  interviews,  travelling  to 
and  from  the  locations  of  the  interviews,  and  calls  back 
when  respondents  arc  not  available  or  do  not  show  up. 
This  .survey  method  probably  should  not  be  used 
unless: 

—  the  effectiveness  measures  data  are  being 
collected  as  part  of  a  larger  data  collection  effort, 
or 

- the  data-  collection  is  done  in  the  information 

;  center.  (This  excludes i, general  population 
.  surveys.)  Center  staff  members  can  conduct 
■  interviews  for.., general  user  surveys,  visitor 

■  surveys,'  or.  -specific  service  surveys.  The 

■  ■■  interviewers  should  •  not  be  those  who  are 

associated  with,  any  specific  service  for  which 
data  are.being  collected  (e.g.,  reported  response 


times,  satisfaction  with  quality,  timelinesSi 
availability,  access,  etc.,  for  specific  services). 

•  In-depth  interviews.  This  method  is  used  for  very 
specific  types  of  evaluation;  for  example,  when  a  very 
new  or  unique  service  is  being  contemplated,  or  for  a 
pretest  of  the  survey.  In-depth  interviews  require  a 
significant  degree  of  interaction  in  order  to  probe  and 
gain  an  in-depth  understanding  of  important  issues. 
The  interviewer  may  actually  follow  a  service  user 
through  the  process  of  interacting  with  the  information 
center  and  its  environment.  When  used  for  a  pretest,  an 
in-depth  interview  can  establish  acceptable 
terminology  that  can  be  used  in  a  survey  questioimaire. 
For  example,  one  can  leam  what  to  call  an  informal 
collection  of  documents  that  is  shared  by  a  group  of 
professionals  (examples  we  have  found  acceptable  in 
different  environments  are  “shared  office  collections," 
“local  library,”  or  “unit  library,”  etc.)  or  what 
professionals  call  regularly  routed  journals  (“routed 
journals,”  “circulated  journals,”  etc.).  One  can  also 
probe  to  gain  a  better  understanding  of  information- 
related  behavior,  general  awareness  of  information 
seiMces,  use  of  altemativi,  ources  (to  the  center)  and 
so  on.  About  six  to  12  such  interviews  can  be 
enormously  helpful.  They  normally  take  30  to  45 
minutes  each.  This  method  is  usually  loosely 
structured,  with  only  a  portion  of  information 
requiring  formal  questions.  Tlie  questions  often  lead  to 
formulation  of  other  issues  and  questions.  The 
interviewer  must  be  very  communicative  and  be  able  to 
identify  when  further  probing  is  necessary.  The  in- 
depth  interview  is  expensive  on  a  per  interview  basis. 

•  Focus  group  interviews.  This  method  involves  gathering 
a  small  group(s)  of  users  or  potential  users  (eight  to  ten 
is  common)  together  in  an  informal  atmosphere  to  get 
them  to  discuss  issues  of  interest.  This  method 
capitalizes  on  interactions  among  the  participants  with 
a  moderate  amount  of  .stimulus  from  a  moderator.  Tlie 
moderator  guides  discussions  and  listens  more  than 
talks.  The  advantage  of  this  method  is  that  interaction 
among  participants  often  creates  a  whole  greater  than 
the  sum  of  the  individual  interaction.  On  the  other 
hand,  the  process  is  expensive  and  yields  information 
from  a  relatively  small  group. 

The  in-depth  interviews  and  focus  group  interviews  are 
often  used  to  enhance  the  communication  performed  on  a 
broad  scale  and  to  formulate  hypotheses  that  can  be  tested 
through  a  statistical  survey. 

7.5  Statistical  Sample  Design 

Statistical  sampling  methods  are  used  for  three  purposes 
First,  they  provide  a  formal  mechanism  for  making  sure  that 
the  most  precise  estimate  possible  are  obtained  for  a  given 
budget.  Second,  they  help  ensure  that  correct  (non-biased  or 
accurate)  estimates  are  obtained.  Finally,  they  provide  a 
formal  means  for  estimating  the  precision  of  the 
observations  made.  Inherent  in  the  statistical  methods  arc 
procedures  for  determining  what  sample  sizes  are  necessary 
for  achieving  certain  levels  of  statistical  precision.  This 
section  discusses  four  important  aspects  of  statistical  sample 
design  and  their  effect  on  the  reliability  of  estimates  based 
on  the  sample  survey.  The  three  principal  kinds  of  estimates 
made  from  evaluation  surveys  are:.. 


62 


•  Popiilaiion  wto/i  including  the  number  of  persons  who 
use  an  information  center  or  its  services,  number  of 
uses  of  the  services,  amount  of  reading  of  journal 
articles  provided  by  information  centers,  total  amount 
of  time  users  spend  in  using  the  services,  etc 

•  Population  averages  including  the  average  number  of 
visits  to  information  centers  per  year  per  user,  average 
time  spent  reading  articles  per  reading,  average  time 
spent  reading  articles  per  reader,  average  rating  of 
satisfaction  with  a  service,  etc. 

•  Population  proportions  including  the  proportion  of  the 
intended  target  population  who  visit  or  use  the 
information  center,  proportion  of  the  population  who 
are  aware  of  a  service,  proportion  of  uses  of  a  service  in 
which  the  user  is  satisfied  or  very  satisfied  with  speed 
of  response,  etc. 

In  addition,  survey  observations  can  and  should  be  made  so 
that  one  can  relate  one  measure  to  another.  For  example, 
one  may  vdsh  to  relate  distance  from  the  center  to  the 
amount  of  use  made  of  center  services,  or  levels  of 
satisi'action  with  amount  of  use  (i.e.,  to  show  that  amount  of 
use  decreases  as  distance  increases  or  satisfaction 
decreases).  We  will  demonstrate  that  one  can  also  obtain 
data  for  sophisticated  modeling  that  illustrates  the  relative 
contribution  such  factors  make  to  amount  of  use  (see 
Chapter  11) 

•  Statistical  sampling  methods.  The  sampling  frame  can 
be  arranged  m  ways  that  can  provide  specific 
information  or  improve  survey  precision  of  estimates 
and/or  cost  effectiveness  of  the  survey  data  collection. 
The  design  also  helps  ensure  that  correct  (unbiased  or 
accurate)  estimates  are  obtained.  They  also  provide  a 
formal  means  for  estimating  precision  of  estimates. 
Examples  of  statistical  sampling  methods  include 
simple  random  sampling,  svstcmatic  sampling  with  a 
random  start,  stratified  random  sampling,  multistage 
sampling,  cluster  sampling  and  ratio  estimation. 

•  Sampling  population  and  sampling  frame.  The 
sampling  population  is  the  universe  of  people,  objects 
or  events  which  one  can  describe  or  measure.  The 
population  may  be  all  professionals  employed  by  the 
center's  parent  organization,  all  scientists  in  the 
country,  the  items  shelved  in  the  center,  or  all  of  the 
uses  on  an  online  catalog.  The  sampling  frame  is  a 
physical  list  of  units  that  make  up  the  population  or 
procedures  that  can  account  for  all  the  sampling  units 
without  the  physical  effort  of  actually  listing  them. 

•  Sample  size.  The  number  of  persons,  objects  or  events 
chosen  for  the  initial  sample.  The  ending  sample  size  is 
the  number  of  units  that  respond  to  the  survey  or  that 
are  observed.  The  size  of  the  sample  (i.e.,  ending 
sample)  partially  determines  the  precision  of  survey 
estimates. 

•  Sample  selection.  Tlie  method  used  for  choosing 
sampling  units  after  a  statistical  sampling  method  is 
established,  and  initial  and  ending  sample  sizes  have 
been  determined. 

Each  of  these  aspects  of  sample  designs  is  discussed  further 
bc'.ow. 

Statistical  Sampling  Methods 

The  most  basic,  straight-forward  sampling  design  is  simple 
random  sampling  although  it  is  infrequently  employed  for 


several  reasons  that  are  discussed  below.  As  the  name 
implies,  the  design  is  based  on  random  sampling  from  a 
sampling  frame  of  individuals,  events  or  objects.  The  basis 
of  i’'c  random  samjile  is  to  be  able  to  assign  a  known 
proi  hility  of  selection  to  each  item  or  unit  in  the  sample 
frame  A  simple  random  sample  implies  that  each  unit  has 
an  equal  chance  of  selection.  For  this  to  be  true,  it  is 
necessary  to  construct  a  sampling  frame  and  establish  a 
selection  procedure. 

Tile  first  step  is  to  define  the  population  to  be  sampled.  A 
population  might  be  all  professionals  in  an  organization,  all 
scientists  who  belong  to  a  professional  society,  all  the  online 
users,  all  online  uses,  or  items  retrieved  from  online 
searches.  It  is  not  enough  to  define  the  population  for 
sampling  purposes;  one  must  also  identify  units  within  the 
population  so  that  they  can  have  a  known  chance  to  be 
chosen  in  a  sample.  The  identity  of  units,  which  are 
frequently  listed,  is  called  a  sample  frame;  and  the  items  on 
the  list  arc  called  sample  units.  The  sample  selection 
procedure  involves  the  manner  m  which  individual  sample 
units  are  chosen.  Some  statisticians  insist  that  the  sample  be 
chosen  in  a  completely  landom  manner,  using  tables  of 
random  numbe-s.  However,  a  perfectly  valid  method,  which 
is  far  more  practical,  is  to  employ  a  systematic  sample  with  a 
random  start.  In  this  procedure  one  simply  takes  the  desired 
sample  size  and  divides  that  number  into  the  number  of 
sample  units  in  the  sample  frame  and  then  samples  through 
the  list  using  that  interval  (called  sampling  interval). 
Suppose  we  have  a  list  (or  directory)  of  2,000  professionals 
in  an  organization.  If  we  wish  to  interview  200  of  the 
professionals,  we  would  divide  2,000  by  200  to  establish  a 
sampling  inicival  of  10.  First  draw  a  random  number 
between  I  and  10  to  use  as  a  starting  point  in  the  sample  — 
say  7.  With  a  sampling  interval  of  10,  the  sampled  units 
would  be  the  seventh  and  every  tenth  name  beyond  that  (i.e„ 
17,  27,  37, . . ,),  The  only  caveat  about  using  a  systematic 
sample  with  random  start  is  that  there  can  be  periodicity  in  a 
listing  which  can  bias  the  sample  selection.  For  example, 
suppose  the  sample  frame  is  days  over  lime.  A  sample 
interval  of  seven  would  mean  that  a  specific  day  (such  as 
Tuesday)  would  always  be  chosen;  and  there  could  be 
differences  among  days  that  would  not  be  represented  in  the 
sample  procedure  described. 

As  mentioned  earlier,  there  arc  statistical  sampling  methods 
that  arc  designed  to  provide  better  results  than  the  simple 
random  sample.  One  of  these  designs,  stratified  random 
sampling,  takes  advantage  of  information  known  about  the 
sampling  units  in  order  to  provide  more  precise  estimates. 
For  example,  if  one  knows  that  some  professionals  are  more 
likely  to  be  center  or  service  users  (e.g.,  scientists  and 
lawyers  versus  financial  and  operational  professionals),  the 
designer  can  apply  this  information  to  improve  the  sample 
design.  That  is  the  sample  frame  can  be  subdivided  into 
groups  of  professionals  of  like  work  roles  (e.g,  R&D, 
operational,  legal  wark,  etc.).  If  the  assumption  concerning 
the  relation  of  work  roh  and  center  or  service  use  is  correct, 
the  precision  of  estimates  should  be  less  than  the  precision 
of  the  estimates  found  from  a  '„inple  random  sample  (or 
systematic  random  sai..p!“)  'i  his  means  that  the  confidence 
iiUeival  of  an  estimate  from  a  stratified  random  sample  is 
narrower  than  estimates  from  a  simple  random  sample,  if  the 
sample  size  is  the  same  for  both  designs.  Conversely,  one  can 
achieve  the  same  level  of  precision  with  a  smaller  sample 
using  a  stratified  random  sample  than  from  using  a  simple 
random  sample.  The  reason  that  the  estimated  precision  of  a 
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Stratified  random  sample  is  less  than  one  estimated  for  a 
simple  random  sample  is  that  the  standard  error  (i.c., 
measure  of  precision)  estimate  is  based  on  the  variability 
within  strata,  which  can  be  substantially  less  than  the 
variability  across  the  entire  population. 

Three  criteria  should  be  kept  in  mind  for  allocating  the 
sample  size  to  the  strata  (c.g,  R&D,  operational,  legal 
professionals,  etc.).  In  a  given  stratum  it  is  usually  best  to 
take  a  larger  sample  if  any  of  the  following  conditions  is  true; 

1 .  Tlie  stratum  total  population  is  larger  than  the  others. 

2.  The  stratum  is  more  variable  (i.e,,  the  range  of 
measures  in  the  stratum  is  greater) 

3.  The  sampling  is  less  expensive  to  conduct  in  the 
stratum. 

In  the  absence  of  appropriate  information  about  the  strata, 
one  will  not  go  too  far  wrong  by  employing  the  same  sample 
fraction  in  each  stratum. 

There  are  exceptions  to  the  rule  above  For  example,  assun.e 
that  an  'ndepcndcnl  information  center  (c  g,  clearinghouse 
or  national  library)  is  surveying  libraries  to  determine  extent 
of  use  of  certain  products  or  services.  Here  the  stratification 
factor  might  be  size  of  the  library.  In  this  case,  it  is  probably 
better  to  sample  more  heavily  in  the  stratum  with  larger 
libraries  (i.e.,  it  is  usually  best  to  give  the  larger  units  a 
greater  chance  of  being  chosen  in  the  sample)  For  example, 
one  would  sum  the  sizes  of  libraries  in  the  large,  medium, 
and  small  strata  as  follows' 


N0.«f 

UbTMtM 

In 

Av«n0«Ste 
^  twfindM 
«f  hoUbpi) 

Sum  oTStm  of 
UbrwiM(OOC9 

StM 

Laiga  libraries 

500 

30 

15,000 

97 

Medium  libranes 

750 

15 

11,250 

72 

Small  libraries 

1,250 

10 

12,500 

81 

Total 

2.500 

38,750 

250 

Thus,  large  libraries  would  be  given  a  higher  probability. of 
being  chosen  in  the  sample  If  libraries  had  been  chosen 
strictly  proportionately,  there  would  be  63  large,  94 
medium,  and  1S6  small  libranes.  Finally,  note  that  if  the 
purpose  of  stratifying  is  to  make  estimates  about  each 
stratum  or  to  compare  estimates  among  strata,  it  is  best  to 
make  the  sample  sizes  about  equal  among  the  strata  (i.c,  in 
the  example  above  the  sample  would  be  84  large,  83 
medium,  and  83  small  libraries). 

In  making  population  estimates  from  stratified  random 
samples  it  is  important  to  take  into  account  the 
disproportionate  sampling  among  strata.  Equations  for 
estimating  totals,  means,  and  proportions  are  discussed  in 
Chapter  10. 

Another  statistical  sample  design  involves  the  situation  in 
which  the  sample  units  of  interest  are  part  of  a  larger  unit. 
For  the  example  of  the  independent  information  center 
above,  suppose  that  the  unit  of  interest  is  online  search  users. 
Here  one  may  want  to  dctcmine  time  spent  searching, 
satisfaction  with  si  arch  results,  or  other  measures.  The 
population  of  all  users  is  difficult  to  identify,  but  libraries 


that  provide  online  search  services  may  be  known.  The 
sampling  method  then  is  to  sample  libraries  (or  information 
centers)  and  subsample  their  online  users.  If  the  number  of 
users  or  online  searches  is  large,  a  subsample  of  them  would 
be  necessary.  This  procedure  is  called  two-stage  sampling: 
the  primary  sampling  units  in  the  example  are  libraries,  and 
the  online  searches  by  users  are  the  secondary  sampling 
units  Sometimes  the  primary  units  are  referred  to  as  clusters 
if  the  number  of  secondary  units  is  not  identified  or  known 
before  the  sample  is  drawn. 

Usually,  to  keep  costs  down,  the  number  of  primary  sample 
units  IS  reduced  and  the  number  of  secondary  sample  units 
is  increased.  The  question  of  how  the  samples  are  allocated 
among  primary  and  secondary  sampling  units  depends  on 
the  vanability  within  clusters.  At  one  extreme,  if  all  the 
secondary  sampling  units  were  exactly  the  same  (e.g.,  if 
libraries  had  rules  that  online  searches  could  be  for  a  given 
length  of  time),  then  it  would  be  necessary  to  choose  only 
one  secondary  sampling  unit  (c.g.,  one  search  per  library). 
The  sample  size  would  be  the  number  of  primary  sampling 
units  chosen.  At  the  other  extreme  is  the  case  in  which  every 
cluster  IS  like  every  other  cluster  (c.g ,  the  average  time  spent 
searching  and  the  variability  of  time  spent  searching  is  the 
same  for  all  libraries)  In  this  instance  one  would  need  to 
sample  only  one  or  a  few  primary  sample  units  and 
subsample  heavily  in  that  unil(s).  The  sample  size  would  be 
the  number  of  secondary  sampling  units  chosen.  Usually  the 
cost  IS  less  for  fewer  primary  units  sampled;  therefore,  a 
balance  must  be  achieved  between  cost  and  precision  of 
estimates.  Generally,  if  information  is  available,  it  is  best  to 
choose  the  primary  sampling  units  on  the  basis  of  size  and 
choose  the  number  of  secondary  units  in  equal  proportion  to 
size.  That  is,  once  a  primary  sample  unit  is  chosen,  a  sample 
frame  is  constructed  for  the  secondary  sampling  units  and  a 
random  sample  is  chosen  with  a  predetermined  sample 
fraction  (say,  every  twentieth  unit).  This  sample  fraction  will 
be  the  same  for  sampling  within  every  primary  sampling 
unit. 

Forming  the  sampling  frame  for  secondary  units  can 
sometimes  be  difficult.  In  the  online  search  example,  there 
may  be  records  of  the  searches  pertormed  over  the  past  six 
months  or  year.  In  this  case,  the  search  time  might  also  be 
recorded  or  the  user  identified,  in  which  case  defining  a 
sample  frame  is  not  a  problem.  However,  if  the  issue  is  to 
establish  users’  attitudes  toward  online  search  services,  it  is 
necessary  to  form  a  unique  list  of  users,  with  recognition  that 
many  users  may  conduct  several  searches  (or  ask  for  several 
searches).  An  even  greater  difficulty  occurs  when 
information  is  needed  from  potential  users  as  well  as  users. 
Here,  potential  users  might  be  selected  from  directories  of 
professionals,  faculty  lists,  enrollment  lists,  or  library 
patrons  lists.  A  combination  of  stratified  ranr  om  sampling 
and  cluster  sampling  is  frequently  used;  that  is,  clusters  arc 
sampled  within  strata. 

There  can  be  any  number  of  stages  in  cluster  sampling.  Foi 
the  example  above,  one  may  wish  to  characterize 
bibliographic  items  found  in  public  libranes,  one  might 
sample  libraries  as  a  first  stage,  bibliographic  publications  as 
a  second  stage,  monthly  issues  as  a  third  stage,  and  items 
within  an  issue  as  a  fourth  stage.  Ii  is  usually  best  to  give  the 
larger  units  a  greater  chance  of  being  chosen  in  the  sample. 
In  fact,  a  common  procedure  is  to  sample  with  probability  in 
proportion  to  size  (pps)  This  is  done  by  listing  the  .sample 
units  within  each  stratum  in  order  of  size  and  selecting  the 
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sample  using  a  systematic  selection  with  random  start.  For 
example,  assume  that  the  stratification  factor  is  the  number 
of  persons  served  (e.g.,  faculty  and  enrollment  in  colleges 
and  universities).  One  would  list  units  in  order  by  their  size 
and  the  cumulative  range  of  sizes  as  follows; 

Cumulative 


Samole  Unit 

S!£S 

Size  Ranoa 

Ubraiy  1 

50,000 

0  -  50.000 

Library  2 

25,000 

50,001  •  75.000 

Library  3 

20,000 

76,001  -  95,000 

Library  4 

1 5,000 

95,001  -  110,000 

Ubraiy  5 

10,000 

110,000  •  120.000 

The  sample  would  be  chosen  by  systematic  sampling,  using 
the  size  information.  The  total  would  be  the  total  cumulative 
size  (120,000  in  the  table  above),  and  this  number  would  be 
divided  by  the  desired  sample  size  (say,  3  libraries)  to 
determine  the  sampling  interval  (i.e.,  40,000).  The  first 
number  would  be  chosen  randomly  between  I  and  40,000 
and  every  40,000th  number  chosen  beyond  that.  If  the 
random  start  was  18,.'i42,  the  subsequent  numbers  would  be 
58,542  and  98,542.  This  means  that  the  sample  of  three 
would  be  Library  1,  Library  2,  and  Library  4,  since  their 
respective  cumulative  size  ranges  include  the  randomly 
selected  numbers. 


from  a  random  sample  and  is  called  standard  error. 
Precision  of  survey  sample  estimates  is  a  function  of 
sampling  error. 

•  NonsampUng  error:  This  error,  sometimes  called  bias, 
IS  that  part  of  the  difference  between  an  estimate  and 
the  true  population  value  is  due  to  mistakes  in  survey 
processes  or  impreciseness  of  survey  communication. 
Nonsampling  error  is  rarely  attempted  to  be  mc.a.sured 
.  because  of  the  extreme  difficulty  of  doing  so.  Accuracy 
of  survey  sample  estimates  is  a  function  of 
nonsampling  error. 

The  total  difference  between  survey  estimates  and  true 
population  values  is  due  to  a  combination  of  these  two  types 
of  errors  in  the  following  way 


Non-sampling 

Error 


Another  statistical  method  that  has  proven  extremely  useful 
in  improving  the  precision  of  estimates  is  miio  estitnanon 
This  method  employs  an  auxiliary  variable  about  which  we 
have  information  in  order  to  calibrate  estimates.  In  this 
method  it  is  assumed  that  the  auxiliary  variable  is  highly 
correlated  to  the  variable  of  interest.  For  example,  we 
indicated  earlier  that  the  amount  of  online  searching 
performed  in  a  library  (X)  is  probably  highly  correlated  to 
the  size  of  a  library,  say  holdings  (y).  We  also  probably  know 
the  total  number  of  holdings  in  a  population  of  libraries  (c.g., 
academic  libraries)  since  directory  listings  usually  indicate 
library  holdings.  Ratio  estimation  is  like  using  a  stratified 
random  sample  wher  j  the  stratification  factor  is  continuous 
rather  than  discrete  (e.g.,  large,  medium,  and  small  libraries). 
In  effect  the  variate  used  in  the  ratio  estimation  merely 
serves  as  a  powerful  calibration. 

7.6  Survey  Reliability 

One  cannot  expect  estimates  of  proportions,  averages, 
totals,  etc.  taken  from  a  survey  to  be  exactly  the  same  as  the 
true  population  value.  The  difference  between  survey 
estimates  and  true  population  values  is  attnbutable  ;o 
survey  errors  There  are  two  types  of  error  that  can  occur  in 
surveys. 

•  Sampling  error:  The  difference  between  an  estimate 
(e.g.,  proportion,  average,  total)  and  the  true 
population  value  which  is  due  to  the  fact  that  only  a 
sample  of  values  is  observed.  If  the  survey  is  a  census 
(i.e.,  every  unit  in  a  population  is  observed)  there  would 
be  no  sampling  error.  Sampling  error  can  be  estimated 


That  IS,  the  total  survey  error  is  like  the  hypotenuse  of  a  nght 
triangle;  where  the  legs  of  the  right  triangle  are  sampling 
error  (x)  and  non-sampling  error  (y).  Thus,  total  survey  error 

IS' 

^/r  +  y' 

Sampling  Errors 

Precision  of  estimates  from  samples  is  measured  by 
standard  errors  of  estimates  Equations  of  standard  error 
include  the  deviation  of  responses  from  the  true  value  and 
sample  size.  Examples  of  equations  of  standard  error  (SE) 
are-as  follows; 

Propornon.  The  equation  for  the  standard  error  of  a 
proportion  SE(p)  is: 


Where:  p  is  the  estimate  of  the  proportion  of  interest 

q  is  the  complement  of  p  (i.c.,  1— p) 
n  is  the  sample  size. 

Referring  to  Table  71,  we  find  that  the  users  rated 
satisfaction  with  relevance  of  search  output  as  follows; 


TABLE  7.1 

SATISFACTION  RATINGS  OF  50  SURVEY 
RESPONDENTS 
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Sample 

Satisfection 

Rating 

Score 

Sample 

Satis^iction 

Rating 

Score 

1 

3 

26 

3 

2 

5 

27 

5 

3 

1 

28 

3 

4 

4 

29 

4 

5 

4 

30 

5 

6 

5 

31 

4 

7 

5 

32 

5 

8 

2 

33 

4 

9 

1 

34 

2 

10 

4 

35 

5 

11 

5 

36 

4 

12 

4 

37 

2 

13 

3 

38 

3 

14 

2 

39 

4 

15 

2 

40 

4 

16 

3 

41 

3 

17 

5 

42 

4 

18 

2 

43 

5 

19 

4 

44 

5 

20 

3 

45 

2 

21 

4 

46 

1 

22 

5 

47 

3 

23 

4 

48 

5 

24 

5 

49 

3 

25 

2 

50 

4 

Rating  Scoroa:  1  •  Vary  Olaaatlafiad,  2  -  Olaaatiafiad,  3  -  Naithar  Satiaflad  nor  Olaaatiafiad,  4  ■  Satlafiad,  5  -  Vary  Sat’adad. 


Satisfaction 

Rating 

Rating 

Score 

No.  of 

Users 

Proportion 
of  Users 

Very  Dissatisfied 

1 

3 

0.06 

Dissatisfied 

2 

8 

0.16 

Neither  Satisfied 

Nor  Dissatisfied 

3 

11 

0,22 

Satisfied 

4 

14 

0.28 

Very  Satisfied 

5 

14 

0.28 

Total 

50 

1.00 
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Thus,  0.56  (56%)  of  the  users  rated  searches  as  satisfied  or 
very  satisfied  (0.28  +  0.28).  Thus,  the  standard  error  of  this 
estimate  is  as  follows: 


-0.07  (7.0%) 


The  confidence  level  for  this  estimated  proportion  is: 
56.0%  ±  1 3.8%  at  95%  level  of  confidence 


Ai’emge.  The  equation  for  the  standard  error  of  an  average 
SE(!!)is: 


SE(x) 


'X(x,-x)^ 

n(n-l) 


Where:  xi  is  one  of  the  observanons 

X  is  the  estimated  average 
n  is  the  sample  size 

X  means  that  (x,-X)^  is  added  up  over  all  x, 


Referring  again  to  Table  7. 1 ,  the  average  satisfaction  rating  is 
found  to  be  3.56  (178  divided  by  50).  The  estimated 
standard  error  of  this  average  is  found  as  follows: 

X(x,-X)2  -  (3-3.56)=  +  (5-3.56)=  + ...  etc. 


Since  observations  (x)  have  only  five  values  (1, 2, 3, 4, 5)  the 
summing  is  simplified  as  follows: 

Z(x,-X)=  -  (1-3.56)=  X  3  +  (2-3.56)=  X  8  +  (3-3.56)=  X 
11  +  (4-3.56)=  X  14  +  (5-3.56)=  X  14  +  (4-3.56)=  X  14 
+  (5-3.56)=  X  14 

-  19,6608  +  19.4688  +  3.4496  +  2.7104  +  290304 

-  74.32 


(x,-x)=- 41,968.82 


SE(X) ' 


/(10,(W  (41,968.82) 


(50)  (49) 


—  41,388  minutes  or  690  hours 

The  confidence  interval  for  this  estimated  total  is: 

11,990  hours  ±  1,350  hours  at  95%  level  of  confidence 

A  further  discussion  is  given  pelow  concerning  the  meaning 
of  confidence  intervals. 


Confidence  Inlenals  of  Estimates  from  Samples 

Standard  errors  of  estimates  can  be  used  to  calculate 
confidence  intervals  of  estunates.  A  common  way  of 
expressing  eonfidence  intervals  is: 

The  proportion  of  users  satined  and  very  satisfied  wtth 
relevance  of  search  output  is: 

56.0%  ±6.1%  at  0.95  level  of  confidence 
or 

(49.9%;62.l%>)  at  0.95  level  of  confidence 

This  means  that  the  confidence  interval  has  a  95  percent 
chance  of  containing  the  true  population  value  of  the 
proportion  of  users  who  are  satisfied  and  very  sansfied 
Confidence  intervals  are  in  fact  esUmates  from  a  sample,  just 
as  a  proportion,  average  or  total  is  an  estimate.  Thus,  if 
another  sample  is  taken  one  would  arrive  at  another 
estimate  to  the  proportion  above,  as  well  as  the  width  of  the 
confidence  interval.  In  fact,  if  one  repeated  the  sample  many 
times,  we  would  find  that  95  percent  of  them  would  include 
the  true  population  value.  This  is  shown  by  the  figure  below: 


SE(x) 


/  74.32 
<  50(49) 


=  0.17 


The  confidence  interval  for  this  estimated  average  is: 
3.56  ±  0.33  at  95%  level  of  confidence 


Total.  The  equation  for  the  standard  error  of  a  total  SE(X)  is: 


SE(X) 


'N^X(x,-x)^ 

n(n-l) 
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Where:  x,  is  one  of  the  observations 

X  is  the  estimated  average 
n  is  the  sample  size 
N  is  the  population  size 
X  means  that  (x,— X)=  is  added  up  over  all  x, 

Table  7.2  gives  values  of  search  times.  The  estunated  average 
search  time  is  71.9  minutes,  if  we  assume  that  there  are 
10,000  total  searches  a  year,  our  estimate  of  total  search  time 
is  719,000  minutes  or  11,990  hours.  The  estimated  standard 
error  of  total  search  time  is  calculated  as  follows: 


Znw 

Itgailatloa 

Vhlue 


As  shown,  most  of  the  confidence  intervals  contain  the  true 
value  which  is  represented  by  the  vertical  line.  The  estimated 
value  (i.e.,  proportion,  average,  total,  etc.)  is  sometimes  to 
the  left  of  the  true  value  line  and  sometimes  to  the  right  and, 
therefore,  the  ennre  interval  moves  accordingly. 
Furthermore,  the  width  of  the  confidence  interval  changes 
for  each  sample. 
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TABLE  72 

VALUES  OF  SEARCH  TIMES  FOR  50  SURVEY 
RESPONDENTS 


Sample 

Time 

(Minutes) 

Sample 

Time 

(Minutes) 

1 

42 

26 

79 

2 

108 

27 

58 

3 

32 

28 

46 

4 

21 

29 

99 

5 

84 

30 

56 

6 

61 

31 

119 

7 

110 

32 

94 

8 

52 

33 

34 

9 

1 03 

34 

79 

10 

72 

35 

96 

11 

31 

36 

59 

12 

89 

37 

39 

13 

63 

38 

114 

14 

112 

39 

74 

15 

51 

40 

97 

16 

68 

41 

39 

17 

84 

42 

107 

18 

63 

43 

79 

19 

33 

44 

58 

20 

85 

45 

97 

21 

92 

46 

120 

22 

115 

47 

37 

23 

48 

48 

118 

24 

97 

49 

93 

25 

46 

50 

47 

68 


The  width  of  the  confidence  interval  is  affected  by  two 
factors: 

•  the  desired  level  of  confidence 

•  the  estimated  standard  error  (SE) 

The  confidence  interval  is  computed  from  the  following 
simple  equation- 

Confidence  interval  is  the  estimate  plus  or  minus  the  standard 
error  times  a  factor  which  represents  the  desired  confidence 
level 
or 

for  example:  P±SExt 

The  value  of  t  is  determined  by  the  desired  level  of 
confidence.  For  example: 

t  —  1.00  for  68  percent  level  of  confidence 
t  —  1.64  for  90  percent  level  of  confidence 
t  ”•  1.96  for  95  percent  level  of  confidence 
t  “  2.57  for  99  percent  level  of  confidence 

Going  from  a  68  percent  to  a  95  percent  level  of  confidence, 
one  would  merely  double  the  width  of  the  confidence 
interval  (i  e.,  1  times  standard  error  to  1.96  times  standard 
error).  This  is  shown  in  the  figure  above  by  x  and  o  on  the 
confidence  interval: 

68%  level  of  confidence 

0  X  X  o 

I _ I 

95%  level  of  confidence 


Ihus,  at  68  percent  level  of  confidence  (shown  by  x,)  the 
width  of  the  confidence  interval  is  narrower  than  the  95 
percent  level  of  confidence  (shown  by  O,)  and  only  68 
percent  of  the  confidence  intervals  would  contain  the  true 
values. 

The  size  of  estimate  standard  error,  m  turn,  is  affected  by 
four  factors.  Assume  example  confidence  intervals  for  an 
estimated  proportion  of  0.50  (or  50%)  with  a  simple 
random  sample  size  of  200  might  be  displayed  as: 

50%  +  3.5%  at  68%  level  of  confidence 

( 1 )  Sample  size.  For  the  example,  the  estimates  above  are 
estimated  from  a  sample  size  of  200  observations.  If 
one  doubled  the  sample  size  to  400,  the  confidence 
interval  would  decrease  from  3.5  percent  to  2.5 
percent  at  68  percent  level  of  confidence.  If  the  sample 
were  reduced  to  100,  the  confidence  interval  would 
increase  from  3.5  percent  to  5.0  percent  at  68  percent 
level  of  confidence 

(2)  Sample  size  relative  to  population  size.  If  the  sample 
size  were  in  fact  the  entire  population,  the  confidence 
interval  would  be  zero.  The  example  above  assumes 
that  the  sample  size  is  very  small  compared  with  the 
population  size.  Since  the  sample  size  of  some  types  of 
users  may  be  reasonably  high  compared  with  the 
population  of  users,  there  is  some  gam  in  reduced 
confidence  intervals.  For  the  example  above,  if  one 
assumes  that  a  sample  of  100  users  of  a  service  is  from  a 
population  of  300  users  the  confidence  interval  would 
fall  from  5.0  percent  to  4.1  percent. 


(3)  Inherent  variability  jf  observations.  For  example,  if 
specified  ratings  of  importance  of  or  satisfaction  with 
interlibrary  lending  and  reference  services  ranges  from 
1  to  5,  the  confidence  interval  for  the  estimated  average 
level  of  performance  would  probably  be  greater  than 
estimates  in  which  ratings  ranged  from  3  to  5. 

(4)  Statistical  sample  method.  A  survey  can  be  improved  in 
terms  of  decreasing  estimated  confidence  levels  (at  a 
given  sample  size)  by  statistical  sample  methods  (e  g., 
stratification,  ratio  estimation,  etc.). 

Examples  of  one  standard  error  (i.e.,  68%  level  of 
confidence)  for  various  sample  sizes  and  estimated 
proportions  (%)  are  given  below,  assuming  a  random 
sample. 


Estimated  Proporttoo 
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57 
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50 
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24 

33 

37 
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21 

28 

32 

35 

35 
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31 

32 
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20 

23 

24 

25 

Note  that  complementaiy  proportions  (i.e,  95%  for  5%. 
70%  for  30%,  etc.)  have  exactly  the  same  standard  errors. 
This  tabic  can  be  used  to  determine  sample  sizes  initially  or 
to  test  approximate  statistical  precision  of  survey  results  For 
example,  if  one  expects  about  30  percent  of  users  to  have 
borrowed  books  on  their  last  visit  and  one  is  willing  to 
accept  estimates  between  25  percent  and  35  percent  at  68 
percent  level  of  confiocnce.  a  sample  size  of  about  85  is 
sufficient  Note,  however,  that  this  means  85  completed 
responses  (not  the  mimbcr  of  questionnaires  handed  out  or 
mailed).  Assume  that  a  sample  of  152  responses  was 
received  and  the  proportion  of  users  who  used  the  card 
catalog  was  estimated  to  be  10  percent.  The  confidence  level 
of  this  e.stimate  would  be  about  plus  or  minus  2.4  percent  at 
the  68  percent  level  of  confidence  or  4.7  percent  at  the  95 
percent  level  of  confidence 

It  IS  important  to  note  that  if  certain  stratum  results  or 
estimates  are  particularly  important  for  stratified  random 
samples,  the  sample  size  of  that  stratum  should  partially 
determine  sample  size.  For  example,  if  it  is  important  to 
know  re.sults  for  R&D  professionals,  the  sample  size  for 
R&D  professionals  should  be  considered  and  the  table 
above  used  as  evidence  for  how  many  of  these  professionals 
to  sample. 


Non-siiniplmg  Error 

Non-sampling  errors  are  mistakes  that  creep  into  survey 
processes  due  to  the  following  sources  of  error. 

•  Improper  questionnaire  design 

•  Development  of  an  inadequate  sampling  frame 

•  Errors  due  to  nonresponse  from  a  .sampled  unit 

•  Errors  in  response  to  questions 

•  Clerical  processing  error 

•  Computing  error 


•  Analyst  error 

•  User  error. 

These  sources  of  error  are  discussed  briefly  below 

One  can  have  very  precise  estimates  (with  narrow 
confidence  intervals),  but  be  measuring  the  wrong  value. 
Thus,  we  would  have  the  picture  below  which  shows 
confidence  intervals  given  above  in  the  section  on  sampling 
errors. 


1 

1  X 

X  1 

1 

1  1 

\ 

1 

1 

1 

1 

I 

I 

1 

1 

1 

1  1. 

1 

1 

1 

I 

1 

_ 

Tnie 

Kymlatfon 

VUve 


Some  examples  of  sources  of  nonsamphng  error  and  means 
of  controlling  them  are  given  below*. 

Questionnaire  design.  The  basic  source  of  errors  m 
questionnaire  design  involves  the  evaluator’s  inability  to 
communicate  what  data  and  information  are  needed  and 
respondents’  inability  to  communicate  responses  accurately. 
Sources  of  error  include  ambiguous  definitions,  poor 
instructions,  use  of  proxy  data,  and  questionnaire  wording, 
format  and  length.  Control  of  these  errors  is  achieved  by 
careful  wording  of  questions,  etc.,  providing  unambiguous 
instructions,  providing  a  glossary  of  terms,  and  doing  a 
pretest  prior  to  final  design  and  data  collection. 

Questionnaire  design  for  all  four  types  of  surveys  is  a  critical 
clement  in  the  chain  of  events  leading  to  valid  and  reliable 
indicators.  Most  surveys  require  a  data  collection 
instrument  or  questionnaire.  Therefore,  procedures  should 
be  followed  to  ensure  that  correct  answers  to  questions  arc 
obtained  and  errors  are  not  made.  Ideally,  questionnaires 
should  be  pretested  by  personal  or  telephone  interview,  if 
possible,  to  discover  respondents’  understanding  of  the 
questions  and  the  instructions  given.  In  addition,  the  pretests 
provide  an  assessment  of  the  duration  of  the  interview  or 
time  necessary  for  respondents.  Also,  pretests  of  self- 
administered  questionnaires  provide  a  means  for  assessi: 
the  response  rates  that  can  be  expected. 

In  designing  survey  questionnaires,  four  basic  rules  should 
be  kept  in  mind: 

•  Ask  only  for  the  minimum  information  required. 

•  Make  sure  that  the  questions  can  be  answered. 

•  Make  sure  that  the  questions  can  be  answered 
tmihfully. 

•  Make  sure  that  the  questions  willhe  answered. 

To  abide  by  these  rules,  the  suggestions  below  should  be 
followed  in  phrasing  questions  and  in  formatting  a 
questionnaire: 


1.  Verbiage.  Questions  should  be  phrased  using  the 
smallest  number  of  words,  everyday  words,  and  words 
that  are  unambiguous  and  make  sense. 

2.  Complex  questions.  TVo  or  more  questions  should  not 
be  disguised  under  one  question  expecting  a  simple 
answer.  For  example,  one  should  avoid  questions  like: 
“On  your  last  visit  to  the  library  did  you  conduct 
bibliographic  searches  using  published  bibliographies 
or  online  searching?  —  Yes/No.”  That  is  three 
questions  in  one,  and  unless  three  response  boxes  are 
set  up,  answers  will  be  meaningless. 

3.  ComplexJinvened  questions.  Quesbons  should  be 
phrased  so  that  if  the  respondent  agrees  with  the 
statement  he  or  she  can  answer  “Yes”  (i.e.,  do  not  use 
negatives  m  such  a  way  as  to  invert  the  question). 

4.  Recall,  li  is  difficult  for  respondents  to  remember 
events  over,  say,  one  month.  The  way  to  avoid  this 
problem  is  to  rely  o’  •itical  incidents,  such  as  “When 
did  you  last  conduct  an  online  search?”  or  “How  many 
online  searches  did  you  conduct  last  month?"  These 
questions  are  more  likely  to  give  reliable  answers  than 
“How  many  times  did  you  conduct  online  searches  last 
year?” 

5  Concrete  facts  versus  opinions.  Answer,  to  questions 
about  what  people  acmally  do  are  likely  to  be  more 
reliable  than  answers  to  questions  about  what  they  like, 
feel,  or  want  for  the  future.  For  example,  quesUons 
about  future  purchase  behavior  or  price  that 
respondents  will  pay  for  an  information  product  or 
service  are  not  very  reliable. 

6.  Use  'critical  incidents’' oi  use  of  services  when  possible 
rather  than  general  statemenit.  For  example.  “What 
was  your  level  of  sabsfaction  with  the  response  time  of 
your  last  online  search?”  is  preferable  to  “Wliat  .s  your 
level  of  satisfaction  with  online  searches?” 

Above  all,  when  designing  questionnaires  it  is  important  for 
the  questionnaire  designer  to  visualize  how  easily  the 
respondent  can  answer  each  question. 

In  physically  organizing  the  questionnaire,  it  is  important 
that  a  question  be  included  only  if  it  is  essential.  Lengthy 
questionnaires,  which  the  respondent  may  not  see  the  need 
for,  tend  to  result  in  thoughtless  answers  and  lower  response 
rates.  It  is  also  important  that  the  first  few  questions  be 
particularly  important  questions,  ones  whose  relevance  can 
be  seen  by  the  respondent.  Extra  care  should  be  taken  with 
the  substance  and  phrasing  of  these  first  few  questions.  In 
order  to  avoid  conditioning  answers  to  subsequent 
questions  by  what  is  asked  in  earlier  ones,  it  is  best  to  go 
from  general  questions  to  specific  questions  than  to  go  from 
the  simple  to  the  difficult  questions. 

Open-ended  questions  are  best  placed  near  the  end  of  the 
questionnaire  so  that  the  closed  questions  will  be  answered 
before  the  respondent  tires.  Bonng  or  repetitive  questions 
especially  should  be  placed  at  the  end  of  a  questionnaire  to 
avoid  respondent  refusal.  The  objective  of  organizing  the 
questionnaire  is  to  provide  a  sequence  that  is  natural  and 
easy  for  the  respondent;  therefore,  topics  and  questions 
need  to  be  arranged  in  the  pattern  which  makes  the  most 
sense  to  the  respondent 

Qne  of  the  biggest  problems  with  surveys  is  refusal  to 
respond,  particularly  with  self-administered  questionnaires. 
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Ways  to  avoid  nonresponse  are  to  make  sure  self- 
administered  questionnaires  are  brief,  have  good 
typographic  quality,  and  demonstrate  interesting  issues. 
Respondents  are  unable  to  understand  the  relevance  of  a 
survey  when  there  arc  inadequate  explanations,  badly 
worded  questions,  and  poorly  ordered  questions.  Personal 
questions  will  achieve  lower  response  rates  and  may 
contribute  to  the  respondents'  refusal  of  the  whole 
questionnaire.  Questions  that  are  not  understood  by 
respondents  will  be  refused.  Finally,  questions  concerning 
attitudes  may  achieve  lower  response  rates  than  more 
straight-forward  factual  questions. 


questions  to  answer,  which  countv.-balances  the  above. 
Regardless,  one  must  be  concerned  with  this  source  of  error. 
An  example  is  given  below  to  demonstrate  the  potential 
effect  of  this  kind  of  error.  Assume  that  we  are  attempting  to 
estimate  the  proportion  of  professionals  in  an  organization 
who  are  familiar  with  or  aware  of  an  information  center 
service.  If  we  assume  an  initial  sample  of  200  professionals 
and  an  ending  sample  of  150  (i.e,  75%  response  rate)  one 
might  have  the  following  results. 

75%  RESPONSE  RATE 

RosnonUenH  (1501  ltetSwpvn!teCB.g)  BW.VS'W 


Almost  every  survey  will  require  that  results  be  classified 
into  certain  groups,  so  that  one  can  see,  for  example,  how 
women’s  behavior  differs  from  that  of  men,  or  how  R&D 
professionals  use  center  services,  or  how  level  of  education 
affects  use  of  services.  The  demographic  characteristics  to 
include  depend  on  the  type  of  survey  being  conducted  and 
which  characteristics  might  be  related  to  difference  in 
behavior. 

Development  of  the  sampling  fmme.  One  major  problem  is 
that  listings  of  i.idividuals  (organizations  or  libraries)  arc 
often  not  complete  or  arc  out  of  date.  For  example,  comp,any 
telephone  lists  are  frequently  out  of  date  or  do  not  make  a 
distinction  of  employees  that  one  must  sample  (e.g., 
professionals).  Thus,  a  •‘target”  population  or  intended 
population  may  not  be  the  one  sampled.  Sometimes,  the 
difference  may  be  judged  to  be  inconsequential.  Otherwise 
steps  should  be  taken  to  update  the  list.  For  example, 
personnel  offices  will  sometimes  provide  names  of  persons 
recently  hired  or  fired.  One  common  error  is  that 
researchers  sometunes  sample  one  kind  of  sampling  unit, 
but  actually  observe  another.  For  example,  information 
center  visitor  or  "exit"  surveys  have  visits  as  the  primary 
sampling  umt;  however,  survey  measures  are  taken  as  though 
the  visitors  are  die  primary  sampling  unit.  This  source  of 
nonsampling  error  can  be  controlled  through  proper 
analysis  and  weighting  methods.  Sometimes  non-obvious 
sampling  frames  must  be  used.  For  example,  m  an 
information  center,  a  sample  of  books  can  be  done  by 
systematically  sampling  catalog  cards  or  linear  distance  on 
shelves  (e.g.,  to  take  books  identified  every  two  linear  feet). 
We  were  recently  asked  bytheLibraryof  Congress  to  design 
a  sample  to  estimate  the  number  of  books  that  require 
special  treatment  for  preservabon.  We  considered  taking 
books  every  “x-th"  feet  on  the  shelves  until  we  found  out  that 
there  are  526  miles  of  shelving.  Instead  we  used  multi-stage 
sampling  with  locations,  shelves  within  locations,  and  books 
within  shelves  serving  as  levels.  Thus,  statistical  sampling 
methods  can  overcome  some  problems. 

Nonresponse  error.  When  surveys  are  done  some  individuals 
refuse  or  neglect  to  respond.  There  are  two  kinds  of 
nonresponses.  One  kind  is  where  a  respondent  refuses  or 
neglects  to  respond  at  all  (i.e.,  unit  nonresponse)  and  the 
other  is  when  the  respondent  doesn’t  answer  a  specific 
question  (i.e.,  item  nonresponse).  This  happens  most  often 
with  self-administered  questionnaires,  less  with  telephone 
interviews  and  least  with  personal  interviews.  The  problem 
is  that  those  who  choose  not  to  respond  to  the  survey  or  to 
specific  questions  may  be  different  from  those  who  do  and, 
therefore,  their  responses  might  be  different.  For  example, 
for  a  survey  about  online  search  services,  users  may  be  more 
likely  to  respond  than  non-users  because  of  interest  in  the 
service.  However,  non-users  usually  would  have  fewer 
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This  example  shows  that  if  the  awareness  of  150 
respondents  is  80.0  percent  and  awareness  of  non- 
respondents  IS  90.0  percent,  the  true  population  value  is 
82.5  percent  (or  2.5%  above  the  survey  estimate).  Even  if  the 
awareness  of  non-respondents  was  half  that  of  respondents 
(an  unlikely  event),  the  true  population  value  would  be  70.0 
percent  (or  10%  below  the  survey  estimate)  which,  for 
decision-making  purposes,  is  not  appreciable  However,  one 
can  sec  as  response  rates  go  down  the  potential  error  will  be 
exacerbated. 

50%  RESPONSE  RATE 
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We  always  attempt  to  achieve  at  least  50  percent  response 
rales.  Again,  as  said  elsewhere,  it  is  belter  to  design  the 
survey  with  fewer  responses  and  devote  resources  to  getting 
a  sufficiently  high  response  rate  than  to  spend  a  lot  on  a  large 
sample  but  achieve  low  response  rates. 


The  principal  methods  used  to  overcome  the  potential  bias 
created  by  non-response  are  imputation  and  weighting. 
Imputation  is  the  process  of  developing  estimates  for 
missing  or  inconsistent  data  in  a  survey.  If  possible,  data 
obtained  from  other  units  in  the  survey  are  usually  used  in 
developing  the  estimate.  Sometimes  a  cohort  is  used.  For 
example,  if  we  have  surveyed  a  large  information  center  staff 
to  determine  how  they  spend  their  time  and  one  staff 
member  did  not  respond  (s.iy,  due  to  an  illness)  one  could 
choose  a  cohort  (i.e.,  someone  who  does  essentially  the  same 
kind  of  work)  and  substitute  the  cohort’s  response  for  that  of 
the  nonrespoiident.  In  a  larger  survey  sometimes  the  overall 
average  is  substituted  for  non-  response.  This  can  also  be 
achieved  by  equally  weighting  all  responses  to  population 
totals.  For  example,  suppose  we  are  estimating  the 


71 


proportion  of  professionals  who  are  visitors  and  the  total 
number  of  visits  that  professionals  make  to  an  information 
center.  Further  assume  that  the  population,  survey  sample, 
and  sample  response  is  500,  300;  and  160,  respectively.  If 
the  number  of  respondents  who  indicate  they  visited  an 
information  center  m  the  past  month  is  107,  we  as.sumc  that 
66.9  percent  (i.c.,  107  divided  by  300)  of  the 
nonrespondents  are  the  same  proportion  of  the  entire 
population  visited  an  information  center.  Similarly,  if  we 
observe  that  the  average  number  of  vi.sits  is  4.88  per 
respondent,  we  can  assume  that  the  average  number  of  visits 
of  nonrespondents  and,  hence,  entire  population,  is  also 
4,88  visits  per  month  per  professional.  However,  we  may 
have  more  information  to  help  impute  or  weight  more 
accurately. 


Suppose  that  the  responses  for  an  organization  can  be 
accurately  subdivided  by  work  role  as  follows. 
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Since  response  rates  vary  substantially  among  work  roles 
(i.e„  from  34%  for  operations  and  other  to  80%  for  R&D) 
and  proportions  and  averages  vary  among  these  work  roles, 
one  can  use  this  information  to  more  accurately  estimate 
proportions  and  averages.  The  estimated  proportion  is 
calculated  by  multiplying  the  work  role  propoition  times  the 
total  in  the  population,  summing  these  totals,  and  dividing 
by  the  population  total  (2,500).  Thus,  the  revised  esbmatc  of 
proportion  professionals  who  visited  an  information  center 
the  past  month  is  70,4  percent  (0.950  X  673  +  0.700  X  978 
+  0.375  X  420  +  0.650  X  429).  This  compares  with  an 
estimate  of  66.9  percent  using  raw  unweight^  results.  Tl’e 
weighted  average  visits  is  5.28  visits  (7.66  X  673  +  5.41  >' 


978  +  2.89  X  420  +  3.58  X  429)  divided  by  2,500.  Tliis 
compares  with  the  raw  average  of  4.88  visits  per  month  per 
profession.ll. 

Response  errors.  .Sometimes  nonsampling  errors  are 
incurred  because  respondents  simply  respond  incorrectly. 
This  can  be  intentional  (e.g.,  lying  about  one’s  age,  time 
spent  in  an  information  center,  etc.)  or  unintentional  (e.g., 
)ust  circled  the  wrong  response,  remembered  incorrectly, 
etc).  Other  reasons  for  unintentional  response  errors 
include  inability  to  answer  (e.g.,  a  respondent  may  not  know 
the  answer  to  a  question)  or  it  costs  too  much  to  answer  a 
question.  Response  errors  tend  to  increase  with  the  length  of 
a  questionnaire.  We  give  some  suggestions  for  minimizing 
response  errors  in  the  discussion  above  concerning 
questionnaire  design. 

Other  sources  of  error.  Nonsamphng  error  can  occur  from 
clencal  processing  of  survey  forms  (e.g.,  sent  to  wrong 
address  or  neglecting  to  input  to  processing),  coding 
responses  incorrectly,  inputting  incorrectly,  or  proofing. 
Selection  and  training  of  clencal  staff  can  go  a  long  way 
toward  minimizing  these  errors.  Also,  review  of  coding, 
editing  routines  and  validating  input  can  reduce  errors. 
Surpnsingly,  computer  errors  can  occur  due  to  improper 
instructions  or  specifications,  rounding  or  truncation,  or 
improper  weighting,  aggregation  or  handling  of  models. 
These  errors  can  be  detected  by  calculating  some  estimates 
manually.  These  kinds  of  errors  are  more  prominent  on  large 
government  surveys,  but  should  be  known  by  evaluators  as 
well.  Errors  can  occur  because  of  improper  analysis  and 
denvation  of  estimates.  Examples  are  given  m  Chapters  8 
and  9  for  overcoming  typical  errors.  Also,  one  can  interpret 
results  invalidly.  The  best  way  to  avoid  these  errors  is  by 
reviewing  analysis  (by  peers,  supervisors  or  an  expert 
panel).  Finally,  the  results  may  be  misinterpreted  by  readers 
(i.c.,  managers)  due  to  inadequate  presentation  Survey 
results  must  include  adequate  description  of  data  attributes, 
survey  methods  employed,  potential  sources  of 
nonsampling  error  and  statistical  precision.  Of  course,  one 
source  of  user  error  is  when  results  are  misused  and  this 
does  happen  on  occasion. 
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Chapter  8 

Evaluating  the  Effectiveness  of  Information  Centers 


8.1  Background 

We  begin  by  refernng  back  to  the  framework  for  evaluating 
information  center  performance  and  effectiveness  in  Figure 
4  in  Chapter  2.  In  the  framework  we  show  that  one  can 
characterize  user's  work  by  the  activities  they  perform  (e.g., 
primary  research,  engineering,  legal  work,  management, 
etc.)  and  resources  they  use  to  do  their  work  (e.g.,  their  time, 
equipment,  support  staff,  information,  etc.).  Tlieir  work  can 
also  be  characterized  by  input  costs  and  output  quantities 
and  attribute.s  (e.q.,  quality  and  timeliness  of  work).  One  can 
relate  the  use  of  information  (in  general)  and  information 
and  services  provided  by  information  centers  (in  particular) 
to  input  costs  and  output  of  users'  work.  We  refer  to  such  a 
relationship  as  higher  order  effects  of  information  and 
information  services.  Evaluating  higher  order  effects  of 
information  and  information  services  is  the  topic  of  Chapter 
10.  This  chapter  focuses  on  measuring  and  evaluating  the 
extent  of  use  of  services  and  factors  that  affect  the  use  of 
services.  Such  factors  include  inherent  need  for  information 
and  services,  awarcnc.ss  of  services,  availability  and 
accessibility  of  services,  satisfaction  with  service  attributes, 
price  of  services,  and  competition  for  service  provision. 
Extent  to  which  information  centers  provide  information 
and  services  is  characterized  by  (D)  and  the  extent  to  which 
service  attributes  affect  use  is  characterized  by  (f)  in  the 
framework  in  Figure  4. 

There  are  three  types  of  surveys  that  can  be  used  'o  measure 
the  use  and  performance  of  information  center  services'*: 

•  survey  of  the  general  user  population  (General  User 
Survey), 

•  survey  of  known  information  center  users  (Patron  or 
Visitor  Survey),  and 

•  survey  of  known  users  or  uses  of  specific  services. 

For  complete  evaluations  we  suggest  conducting  at  least  two 
of  these  surveys.  The  first  two  types  of  surveys  arc  described 
below.  The  surveys  involving  specific  services  arc  discussed 
in  the  next  chapter. 

8.2  General  User  Survey 

The  purpose  of  the  general  user  survey  is  to  obtain 
information  about: 

•  number  of  users  and  extent  of  use  of  the  information 
center, 

•  number  of  users  and  extent  of  use  of  specific  services, 

•  importance  of  and  satisfaction  with  specific  services, 

•  awareness  of  specific  services, 

•  satisfaction  with  staff  competence  (knowledge,  skills, 
and  attitude), 

•  suggestions  for  improvement  in  services  and  service 
attributes,  and 

•  relevant  user  charactenstics  and  demographics. 

This  survey  involves  a  statistical  sample  from  the  entire 
population  of  the  user  community  being  served  such  as  the 


employees  of  a  company  or  government  agency,  specific 
types  of  professional.'  such  as  scientists  and  engineers,  or 
potential  organizations  and  their  employees  served  by  a 
clearinghouse.  The  reason  that  the  entire  population  is 
surveyed,  rather  than  just  users,  is  that  by  doing  so  one  can 
obtain  valuable  information  about  non-users  and  the 
reasons  they  do  not  use  the  information  center  or  specific 
services. 

Suggestions  for  sample  design  are  as  follows: 

•  Sample  frame.  In  a  company  or  government  agency  the 

personnel  department  can  usually  provide  a  listing  of 
employees,  sometimes  categorized  by  useful 
information.  We  often  limit  the  sample  frame  to  certain 
population  segments  that  are  highly  likely  to  be 
information  center  users.  For  example,  often  only 
professionals  In  organizations  are  sampled  This 
excludes  non-professionals  such  as  secretaries, 
laboratory  technicians,  clerical  staff  and  other  tyfies  of 
support  staff.  It  is  important  to  recognize  that  non- 
professionals  sometimes  use  libraries  to  support  their 
work  and,  if  they  are  excluded,  estimates  of  use  will  be 
understated.  Secondly,  support  staff  often  use  an 
information  center  on  behalf  of  professionals.  This 
potential  survey  weakness  can  be  addressed  by  the  way 
111  which  use-related  questions  arc  asked  (see  the 
section  on  questionnaire  design  below). 

If  an  information  center  serves  a  general  community 
(e  g.,  an  information  clearinghouse  or  national  library), 
the  sampling  frame  may  have  to  be  developed  from  one 
of  several  sources.  A  common  source  is  directories  of 
organizations  or  libraries  in  organizations.  In  the  U  S., 
for  example,  one  can  use  listings  (or  samples)  of  firms 
provided  by  Dun  &  Bradstreet,  American  Library 
Association,  Special  Library  Association,  and  other 
directories  of  company,  agency,  academic  institution, 
and  other  organization  libraries  and  information 
centers.  The  sampling  frame  can  also  be  developed 
from  listings  of  individuals  such  as  members  of 
professional  societies.  When  multiple  listings  are  used 
one  must  be  careful  to  take  into  account  organizations 
or  individuals  who  appear  on  more  than  one  list, 
otherwise  their  responses  will  be  biased.  (See 
discussion  under  sample  design  below). 


•  Sample  design.  For  sampling  employees  in 
organizations,  we  find  that  there  is  usually  sufficient 
information  provided  by  personnel  listings  t.hat  one 
'■an  stratify  the  sample  to  improve  the  precision  of 
survey  results.  That  is,  stratification  takes  into  account 
factors  that  are  related  to  extent  of  use  of  services.  Such 
factors  include; 

—  Location  If  an  organization  has  multiple  sites,  it  is 
useful  to  stratify  by  sites  that  do  and  do  not  have 
information  centers. 
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—  Level  of  employee.  As  mentioned  above,  if 
classification  is  available,  it  is  useful  to  stratify  by 
professional,  paraprofessional  (e.g.,  laboratory 
technician,  executive  secretary  and  clerical)  or 
other  support. 

—  Work  role.  Some  organizations  can  quite  easily 
categorize  their  employees  by  work  role  such 
as  research,  engineering.  management, 
administrative,  operational,  etc. 

If  the  purpose  of  stratification  is  to  present  data  and 
analysis  by  the  stratibcation  factors,  then  one  should 
sample  equally  from  each  strata.  If  the  purpose  of 
stratification  is  to  improve  precision,  then  sampling 
from  the  strata  should  be  roughly  in  proportion  to  the 
expected  amount  of  use.  An  example  is  given  below  for 
a  two-way  stratification  where  one  strata  factor  is  work 
role  (sampled  equally)  and  level  of  employee  (sampled 
in  proportion  to  likely  use).  In  the  example,  we  give 
population  and  sample  sizes  (N/n). 


POPUUTION  AND  SAMPLE  SIZE  FOR  TWO-WAY  STRATIFICA'nON 
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If  the  sampling  frame  is  of  organizations,  then  a  two- 
stage  sample  might  be  necessary.  By  two-stage  we 
mean  that  organizations  (e.g.,  library)  is  sampled  as  the 
first  stage  and  end-users  or  individuals  (eg., 
professionals)  in  the  organization  are  sub-sampled  as  a 
second  stage.  The  second  stage  or  individual  sample 
can  be  done  by  having  a  designated  person  in  the 
organization  select  a  sub-sample  of  users  or  persons 
found  in  their  organization.  A  librarian  is  a  good 
person  to  do  the  sub-sampling.  Normally,  one  would 
sub-sample  in  rough  proportion  to  extent  of  use, 
however,  because  of  the  burden  to  the  de.signatcd 
sampler  we  normally  have  them  choose  the  names  of 
3—5  persons  and  either  submit  the  names  to  the 
researcher  to  survey  or  have  the  sampler  forward  the 
questionnaire  to  the  sampled  persons  to  be  returned  to 
the  researcher.  One  can  also  ask  the  librarian  to  send  a 
telephone  directory  of  the  organization  or  3  or  5  pages 
chosen  randomly  from  such  a  directory  and  then  the 
resea  roher  can  choose  a  more  proportional  sample  and 
not  burden  the  librarian  further. 

As  mentioned  above,  one  problem  anses  when  a 
sample  is  drawn  from  listings  in  which  individuals  (or 
organizations)  can  appear  on  more  than  one  of  them. 
Suppose,  for  example,  one  has  three  lists  from  which  to 
sample-  a  list  of  laiown  users,  a  list  from  the  American 
Chemical  Society  (ACS)  and  a  list  from  American 
Institute  of  Physics  (AIP).  Sampled  individuals  could 
possibly  be  on  any  of  the  following  combinations  of 
lists  (with  number  respondents  given  in  parentheses); 
—  User  only  (30) 


—  ACS  only  (15) 

—  AIP  only  (25) 

—  User,  ACS  (66) 

-User,  AIP  (50) 

-ACS,  AIP  (25) 

-User,  ACS.  AIP  (10) 

These  combinations  can  be  considered  seven  strata 
and  treated  as  a  stratified  random  sample,  only  the 
strata  are  established  after  responses  come  in  (i.c., 
post-stratified  sample).  First,  one  must  estimate  the 
population  totals  from  each  stratum.  This  can  be  done 
by  random  sampling  from  each  list  and  then  estimating 
strata  totals  by  determining  the  identity  of  the  lists  each 
respondent  belongs  to.  This  can  be  done  by  cross¬ 
checking  lists  or  by  having  the  respondents  indicate  if 
they  use  the  service  and  are  members  of  ACS  and/or 
AIP.  Assume  that  list  sizes  and  sampled  respondents 
are  as  follows:  users  —  4,000/150;  ACS  —  32,000/ 
100;  AIP  —  18,000/120.  Note  that  only  certain 
sections  of  ACS  and  AIP  might  be  sampled.  The  way  in 
which  the  sampled  respondents  are  partitioned  is  given 
in  the  parentheses  for  each  of  the  seven  combinations 
above.  Hough  equations  for  estimating  strata  totals 
(combinations)  are  as  follows: 

User,  ACS  N(UC)  -  (4,000  X  SO  +  32,000  X  5) 
divider*  by  250  ■«  1,440 

User,  AIP  N(UI)  -  (4,000  X  60  -b  18,000  X  10) 
divided  by  270-  1,560 

ACS,  AIP  N(CI)  -  (32,000  X  25  +  18,000  X  55) 
divided  by  220  -  8,140 

User,  ACS,  AIP  N(UCI)  -  (4,000  X  10  ■+  32,000  X  1 
+  18,000X2)  divided  by  370 -290 

User  only  N(C)  -  (4,000-1,440-1,560-290)  -  710 

ACS  only  N(C)  -  (32,000-1.440-8,140-290)  - 
22,130 

AIP  only  N(l)  -  (18,000-1.560-8,140-290)  - 

8,010 

More  complex  and  accurate  equations  can  be  used,  but 
the  equations  above  arc  adequate  approximations 
Once  totals  for  the  seven  strata  arc  estimated, 
statistical  weighting  should  be  applied  for  all  estimates 
just  as  done  for  stratified  sample  surveys.  (See  Chapter 

n 


•  Sample  size.  The  sample  size  should  be  determined  by 
the  level  of  precision  required  for  the  survey.  One  can 
refer  to  the  table  in  Chapter  7  for  an  indication  of  level 
of  precision  achieved  at  different  sample  sizes.  There 
arc  some  aspects  of  sample  size  that  should  be 
remembered; 

—  The  relevant  sample  size  is  the  actual  responses 
and  not  initial  sample  (i.c.,  mailout  or 
respondents  sampled).  One  should  attempt  to 
achieve  at  least  50  percent  response  rates  and, 
therefore,  if  a  sample  of  1 50  is  desired,  one  "^hould 
sample  around  300.  .Also,  questionnaire '  ^ponse 
rate  is  often  higher  than  item  (i.e.,  question) 
response  rate  because  som»que$tions  will  not  be 
answered  by  some  respondents. 
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—  If  one  is  particularly  concerned  about  a  subset  of 
respondents  (e.g.,  research  work  role  or  persons 
at  remote  site)  the  relevant  sample  size  refers  to 
those  subsets.  One  should  attempt  to  have  at  least 
15  to  25  responses  in  relevant  subsets. 

Generally,  sample  size  can  comfortably  be  about  200 
responses  for  most  user  surveys.  Note  that  some 
information  gathered  in  the  General  User  Survey  (e.g., 
extent  of  use  of  information  center,  work  role)  may  be 
combined  with  other  survey  results  so  that  the 
combined  survey  sample  size  is  the  one  that  should  be 
considered  appropriate. 

•  Random  Sampling  Procedures.  Assuming  that  lists  can 
be  obtained,  the  best  method  for  sample  selection  is 
normally  systematic  sampling  with  random  start.  That 
IS,  one  merely  needs  to  determine  a  sampling  interval, 
select  the  first  sample  randomly  and  then  take  every 
nth  unit  on  the  list.  For  example,  referring  to  the  table 
above  giving  population  and  sample  size  for  two-way 
stratification,  there  arc  950  professionals  who  do 
research  and  we  want  to  sample  60  of  them.  The 
sampling  interval  would  be  16  (950  divided  by  60)  so 
that  a  number  between  one  and  16  would  be  chosen 
from  a  table  of  random  numbers.  If  the  number  is  four 
the  sample  would  be  4, 20  (4  -I- 16),  36  (20  +  16),  etc. 

•  Data  Collection  Methods.  This  survey  would  normally 
be  done  by  mad  and  self-administered,  but  it  could  also 
be  done  by  telephone  or  personal  mterview.  The 
advantages  and  disadvantages  of  these  three  data 
collection  methods  were  discussed  in  the  previous 
chapter. 

8.3  Amount  of  Use  of  Information  Centers 

Amount  of  use  of  information  centers  is  a  critical  measure 
because  it  is  an  indicator  of  the  usefulness  of  them  and  it 
serves  as  a  means  of  diagnosing  performance  of  services.  Of 
course,  the  best  means  of  measuring  use  of  internal 
information  centers  is  by  gate  counts  and  logs  of  other  kinds 
of  contacts  such  as  by  calling,  writing  and  so  on.  However, 
most  information  centers  do  not  keep  such  information  on  a 
regular  basis.  A  second  way  to  measure  use  is  through  a 
sample  of  visits  and  uses  through  other  means.  This  method 
is  discussed  in  the  next  chapter.  A  third  method  is  to 
determine  amount  of  use  through  a  General  Population 
Survey.  In  this  survey  one  should  survey  the  extent  of  use  of 
alternative  sources  to  information  centers  in  addition  to  the 
information  center  or  centers  being  evaluated.  For  example, 
within  a  company  or  agency  there  may  be  a  main 
information  center  and  centers  located  at  two  additional 
sites.  In  addition  to  these  three  centers  of  interest  there  may 
be  "local”  office  collections  of  books,  reference  materials 
and  journals.  These  collections  may  be  maintained  by  a 
secretary,  laboratory  technician  or  other  support  staff.  We 
have  found  that  such  collections  are  valuable  resources  ihat, 
along  with  persona!  subscriptions  to  journals  and  other 
materials  as  well  as  information  centers,  can  achieve 
optimum  allocation  of  materials  across  the  entire 
organization  (see  Chapter  12  for  further  discussion  of  this 
issue).  Alternatives  to  information  centers  in  an 
organization  aie  external  iibranes  and  information  centers 
such  as  academic,  public  and  government  Iibranes  and 
government  clearinghouses. 

Ability  of  respondents  to  remember  their  number  of  uses  in 
the  past  (i.e.,  recall)  is  a  problem  with  asking  about  amount 


of  information  center  use.  Therefore,  the  length  of  penod 
over  which  use  is  determined  depends  on  how  much  use  is 
made  of  information  centers.  If  use  is  high  the  penod  could 
be  one  week.  That  is,  one  would  ask  how  many  times  the 
information  center  was  used  in  the  last  week.  If  use  is  low  the 
period  could  be  as  long  as  a  year.  We  have  found  that  one 
month  is  best  for  most  organizations.  The  problem  with 
using  a  penod  less  than  a  year  is  that  the  “last"  week  or  the 
“last"  month  may  not  be  typical  if  there  are  seasonal  affects 
such  as  are  common  with  academic  institutions  or  in 
December  holidays.  Thus,  the  survey  should  be  done  at  a 
time  that  will  yield  accurate  results  when  the  week  or  month 
period  is  extrapolated  to  a  year. 

There  arc  two  basic  ways  of  asking  respondents  for  their 
amount  of  use.  One  is  by  pre-coded  ranges  of  amount  of  use 
and  the  other  is  by  having  respondents  record  their 
recollection  of  approximate  amount  of  use.  We  have  found 
that  one  should  ask  for  “approximate"  amounts,  because 
some  respondents  (particularly  some  scientists  and 
engineers;  will  be  "tumed-off’  by  the  survey  because  they 
will  not  know  exactly  how  many  times  they  used  the  center 
Also,  we  have  found  that  it  is  useful  to  ask  for  both  number 
of  visits  to  the  information  center  and  additional  uses  such 
as  by  calling,  writing,  etc.  Finally,  it  is  important  to  specify 
that  the  uses  are  for  the  respondents  own  work-related 
activities  and  that  the  number  of  uses  should  include 
instances  where  someone  else  (e.g.,  a  secretary  or  other 
support  staff)  uses  the  information  center  at  the 
respondent’s  request  or  on  behalf  of  the  respondents. 

Examples  of  how  this  question  might  be  asked  are  as  follows: 
Q.  Approximately  how  many  times  in  the  past  month  have 
you  or  someone  on  your  behalf  visited  information 
centers,  libraries,  or  local  office  collections  for  your 
work-related  purposes  and  how  many  additional  times 
have  you  used  them  by  calling,  writing,  etc.  to  obtain 
information,  order  documents,  or  other  purpose’’ 
Include  visits  or  additional  uses  Ihat  are  made  by 
someone  else  (e.g.,  secretary  or  other  support  staff)  at 
your  request  of  or  on  behalf  of  someone  else.  If  you  did 
not  use  an  information  center,  library  or  "local”  office 
collection  in  the  LAST  YEAR,  please  cheek  here  and 
skip  to  Section  X  on  page  Y. _ Go  to  page  Y. 
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Q.  Approximately  how  many  times  in  the  past  month  have 
you  or  someone  on  your  behalf  visited  information 
centers,  libraries  or  local  office  collections  for  your 
work-related  purposes  and  how  many  additional  times 
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have  you  used  them  by  calling,  writing,  etc.  to  obtain 
information,  order  documents,  or  other  purpose? 
Include  visits  or  additional  uses  that  are  made  by 
someone  else  (e.g,,  secretary  of  other  support  staff)  at 
your  request.  Do  not  include  visits  or  additional  uses 
you  made  at  the  request  of  or  on  behalf  of  someone 
else.  If  you  did  not  use  an  information  center,  library  of 
“local”  office  collection  in  the  LAST  YEAR,  please 

check  here  and  skip  to  Section  X  on  page  Y. _ Go 

to  page  Y. 
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Some  additional  notes  concerning  this  question.  It  is 
important  to  establish  acceptable  terminology  for  this 
question  during  initial  in-depth  interviews  (sec  Chapter  7 
for  a  discussion  of  this  method).  This  applies  to  names  and/ 
or  locations  of  information  centers  and  libraries.  Also, 
“local”  office  collections  might  be  more  commonly  called 
“unit  collections,”  or  “laboratory  reading  rooms”  in  an 
organization.  The  skip  pattern  is  used  to  avoid  the  necessity 
of  requiring  non-users  to  answer  questions  concerning  use. 
One  can  make  provision  through  the  skip  pattern  to  avoid 
irrelevant  questions.  We  assume  that  someone  who  hasn’t 
used  an  information  center  or  library  in  the  last  year  can  be 
considered  a  non-user.  Finally,  one  can  refine  estimates  of 
total  use  over  an  entire  year  by  asking  respondents  the 
number  of  days  they  were  at  work  in  the  last  month  to 
account  for  sick  leave,  holidays,  or  vacation.  The  responses 
can  be  adjusted  by  the  following  means: 

•  Assume  13  days  worked  in  the  month  reported,  a 
response  of  6  items  the  last  month  reported,  and  220 
normal  work  days  (without  sick  leave,  holidays,  etc.)  in 
a  year. 

•  Adjustment  to  an  annual  total  would  be  220  divided  by 
13X6“  101.5  times  per  year. 

Thus,  for  this  measure  the  respondent  is  estimated  to  visit 
the  information  center  (or  other  facility)  101.5  items  in  a 
year.  This  value  would  be  imputed  to  the  respondent  and 
analysis  performed  using  these  imputed  data. 

Average  number  of  visits  (or  additional  uses)  would  be 
found  by  summing  annual  estimates  of  responses'  divided 
by  number  of  respondents  (including  non-u.sers  who  skip  to 
later  questions  because  they  did  not  use  an  infcmation 
eenter  in  the  last  year).  Total  visits  is  calculated  by 
multiplying  average  visits  per  person  times  the  total  number 
of  persons  in  the  population  sampled. 

Average  and  total  visits  (or  additional  uses)  calculated  from 
pre-coded  ranges  is  less  precise,  but  can  be  .done  by  using 
the  following  calculations  (assuming  one  month); 
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The  data  above  are  actual  data.  The  mean  of  an  interval  can 
be  approximated  by  the  mid-point  (e.g.,  1, 2,  “  1.5;  3—5  “ 
4.0;  etc.)  or  by  using  a  geometric  mean'  ealculated  by  taking 
the  square  root  of  the  product  of  the  end-points  of  the  range. 
For  example,  the  geometnc  mean  of  the  1, 2  range  is  square 
root  of  1  X  2  “  1.41 ;  3—5  range  is  square  root  of  3  X  5  ” 
3.87;  etc.  The  estimated  mean  of  the  data  above  25  visits  is 
calculated  from  actual  reported  data.  The  overall  mean  is 
found  by  summing  the  cross  products  of  number  of 
responses  times  esti.nated  means  for  each  range  and 
dividing  by  the  total  number  of  responses.  Average  annual 
use  is  then  found  by  multiplying  the  monthly  average  by  12. 
Tile  estimated  average  for  the  data  above  is: 

( .30  X  0  -I-  .22  X  1.41  +  .19  X  3.87  -I-  .15  X  7,75  +  .10  X 
16.58  +  .04  X  32  65)  X  12  -  62.06 

One  particularly  important  factor  involving  the  extent  to 
which  an  internal  information  center  is  used  is  the  distance 
of  users  to  the  center.  This  is  a  measure  of  accessibility  of  a 
center  and  its  services.  Distance  can  be  measured  in  several 
ways  including: 

•  linear  distance  such  as  number  of  feet  (or  even  miles) 
that  a  user  is  to  an  information  center, 

•  number  of  floors  away  (if  appropnaie),  or 

•  number  of  minutes  away. 

We  have  found  that,  by  far,  the  best  measure  is  in  lime  (i  c , 
minutes  away)  because  time  of  professionals  is  a  scarce 
resource.  They  are  very  aware  of  ihcir  time  and  how  much 
time  It  takes  them  to  get  to  an  information  center  In  a  real 
sense  the  time  users  spend  going  to  an  information  center 
and  in  using  its  services  is  a  measure  of  the  “price”  they  pay 
for  the  center  or  the  “value"  they  place  on  the  information 
and  services.  As  the  lime  (distance)  to  the  center  is  increased 
the  use  decreases  because  the  “pnee”  increases.  Just  as 
demand  for  consumer  products  decreases  as  price 
increases,  the  use  of  information  centers  decreases  as  its 
"price”  increases.  Methods  for  measuring  distance  to 
information  centers  are  discussed  below. 

Distance  (m  time)  can  be  asked  in  the  demographic  section 
of  the  questionnaire  or  it  can  also  be  added  to  the  question 
concerning  number  of  visits  and  number  of  additional  uses. 
Examples  are  given  below  for  these  two  methods. 

0  Approximately  how  far  (in  minutes)  is  your  officeJIab 
from  the  Information  Center? 

_ Minutes 


OR 


'  By  mulu'plying  responses  by  12  or  a  refined  value  as  calculated  '  Geometnc  mean  is  a  good  approximation  for  log-normally 
above.  distributed  numbers  (i.e.,  highly  skewed  numbers). 
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Note  that  we  did  not  include  a  skip  pattern  for  non-use 
because  it  may  be  that  a  long  distance  to  an  information 
center  results  in  non-use.  This  hypothesis  can  only  be 
examined  by  comparing  distance  of  users  with  distance  of 
non-users. 

We  find  that  professionals  generally  will  use  a  variety  of 
sources  for  their  information  and  information  services  and 
this  is  as  it  should  be  since  an  organization's  information 
center  should  not  be  expected  to  fulfill  all  information 
needs.  However,  if  over  alwut  10  to  15  percent  of  visits  and 
additional  uses  of  a  center’s  potential  patrons  come  from 
external  sources  (e.g ,  academic  libraries)  there  may  be  a 
problem.  The  reason  that  we  say  there  may  be  a  problem  is 
that  non-organization  informat'.on  sources  tend  to  be 
expensive  in  terms  of  users’  time  Some  use  is  expected;  for 
example,  an  employee  may  be  in  night  school  and  may  ute 
the  academic  library  for  worK-related  purposes  or  a 
professional  may  know  about  a  book  that  is  in  the  public 
library,  etc.  However,  extensive  external  use  can  be  very 
expensive  to  an  organization.  Extensive  use  of  alternative 
sources  also  suggest  that  there  may  be  a  problem  with  the 
performance  of  the  information  center.  There  are  many 
factors  that  affect  use  of  an  information  center  and  if  certain 
use  enteria  are  not  met  by  the  information  center, 
professionals  will  seek  alternative  source. 

Factors  That  Affect  Use  of  Informatton  Centers 

Factors  that  affect  use  of  information  centers  and  their 
services  include; 

•  inherent  need  for  information  and  services, 

•  availability  of  information  and  services  at  the 
information  center  and  alternative  (competitive) 
sources, 

•  awareness  of  center  and  its  services, 

•  importance  of  services,  and 

•  performance  of  information  center  services. 

In  this  section  we  provide  examples  of  how  one  can  measure 
these  factors  by  surveys  and,  in  Chapter  11,  we  show  the 
relationship  of  these  factors  and  extent  of  use  of  the 
information  center  and  its  services.  First,  we  list  examples  of 
information  services  and  then  examine  how  to  measure  use 
of  the  services  and  the  factors  above  the!  affect  use. 

Information  center  services  are  grouped  as  follows; 

•  /Icceij  to  information  -enter  facilities:  Research  and 
reading  roofns,  study  carrels,  disabled  facilities,  etc. 

•  Access  t'j  collections  and  materiah:  Access  to  library 
books,  journals,  patent  documents,  reference 
iiiaicnals,  internal  techmeal  reports  and  other 
documents,  external  documents  such  as  government 
reports,  audio  visual  materials,  cartographic  material 
etc;  journal  routing;  access  to  journal  articles  through 
interlibrary  loan  or  document  delivery;  centralized 


purchase  of  books,  journal  subscriptions,  piofessional 
society  membership,  etc. 

•  Access  to  information  equipment  and  systems:  Online 
bibliographic  and  numeric  database  searching  systems 
available  to  users,  online  catalogs,  terminals  or  PCs, 
audio-visual  equipment,  microform  viewing 
equipment,  photocopy  equipment,  etc. 

•  Reference  and  referral  systems:  In-depth  research, 
search  cf  reference  materials,  online  bibliographic  and 
numeric  database  searching,  online  searching  of 
internal  documents,  referral  searches,  etc.;  current 
awareness  services;  selective  dissemination  of 
information  (SDl);  and  so  on. 

•  Other  services:  Document  translation,  archives  or 
organization  records,  internal  document  publishing 
and  distribution,  organization  electronic  bulletin 
board,  orientation  and  training,  etc. 

This  list  of  services  is  not  intended  to  be  exhaustive,  but 
rather  indicative  of  the  kinds  of  services  that  might  be 
evaluated. 

One  can  ask  a  senes  of  questions  for  each  service 
concerning: 

•  extent  of  use  of  the  service  and  alternative  sources  to 
the  service, 

•  awareness  of  services, 

•  availability  and  accessibility  of  services, 

•  importance  of  services,  and 

•  satisfaction  with  attnbutes  of  output  such  as  quality 
and  timeliness  of  services. 

However,  if  evaluation  involves  a  substantial  number  of 
services  such  a  scries  of  questions  can  make  the 
questionnaire  unacceptably  long.  Instead  these  questions 
can  be  grouped  as  demonstrated  below. 

The  series  of  questions  might  involve  awareness  and  use  of 
sersices.  An  example  of  a  survey  question  concerning  these 
factors  is  given  be)'  '. 

Q.  The  question  below  deals  with  your  awareness  and  use 
of  Information  Center  services.  Even  though  you  may 
be  an  Information  Center  user,  you  may  not  be  aware 
of  or  use  all  of  its  services.  A  list  of  services  are  given 
below. 

Please  indicate  (by  circling  appropriate  numbers)  if 
you  are  aware  of  the  service  (and  have  used  it  —  1  or 
have  never  used  it  —2)  or  that  you  are  not  aware  of  the 
service  (and  have  no  need  for  it  —  3  or  have  a  need  for 
it  —  4).  Also,  record  the  number  of  times  you  have  used 
the  service  in  the  last  month  and  the  number  of  times 
you  have  used  another  source  for  the  service  in  the  last 
month  (c.g.,  an  academic  library,  a  “local”  office 
collection;  etc.). 
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One  fan  be  more  specific  for  some  services.  For  e;  ample, 
for  me  Question  “read  O’  used  collection  of  books  journals, 
etc.  m  the  Center”  couK  'C  subdivided  into: 

•  Read  or  used  collection  of  materials  in  the  Center: 


Not  Aware  of  Soivlea 


Ham  HcvaNmr  Hav«no  Hww 
UMd  UMd  NMdfcr  NMd«w 

8«v«m  Q«vk» 


fASTMOhTW 

MoftlW^OO  OdMT 

CanMr  Soure* 


S£BU£E 


Currer4  period>csis  t 

2 

3 

4 

Current  boons  1 

2 

3 

4 

_ 

_ 

Psnodiceiein 
oonpecl  storspe  1 

2 

3 

4 

_ 

Books  in  comped 
storsge  t 

2 

3 

4 

— 

_ 

Liboretory 

notebooks  ^ 

2 

3 

4 

— 

_ 

Orgeni2Stion  techr^ 
reports  1 

2 

3 

4 

Relsrenee  meterMs  i 

2 

3 

4 

— ■  - 

-  ■ 

AudtoAifsuai 

msteriets  i 

2 

3 

4 

_ 

_ 

Csrtogrephie 

metedsis  1 

2 

3 

4 

Also,  if  the  list  of  services  is  felt  to  be  too  lengthy  i«  is 
possible  to  sub-divide  the  services  and  actually  conduct  two 
surveys  for  two  lists  of  services. 


Information  and  measures  provided  by  the  above  questions 
include; 

•  Number  of  persons  who  use  the  service 

•  Number  of  persons  who  are  not  aware  of  the  service 
and,  more  importantly,  who  also  have  a  need  for  the 
service.  This  provides  and  indieator  of  how  much 
additional  use  might  be  created  through  publicity  or 
marketing.  By  linking  (1)  those  not  aware  but  have  a 
need  to  (2)  Center  users/non-users  and  (3) 
demographics  such  as  location  one  has  strong  evidence 
lo  focus  publicity  to  appropriate  market  segments. 

•  Number  of  uses  (output)  of  services  and  the  extent  to 
which  alternative  sources  are  used  for  obtaining  the 
services. 

Number  of  uses  of  services  can  be  related  to  importance  and 
performance  of  services  and  output  attributes. 

Calculations  for  the  factors  above  arc  straight-forward.  The 
awareness  of  services  can  be  done  as  a  proportion  since 
respondents  must  circle  I,  2,  3,  or  4.  The  proportion  of 
respondents  who  circle  1  gives  an  estimate  for  the 
proportion  of  users  of  that  service.  The  estimated  total 
number  of  users  is  calculated  by  multiplying  this  proportion 
times  the  number  of  professionals  in  the  sampling  frame. 
The  proportion  of  respondents  who  circle  3  and  4  gives  an 
esdmate  of  the  proportion  who  are  unaware  of  the  service 
and  4  gives  the  proportion  who  are  unaware  and  have  a  need 
for  the  service.  Inherent  amount  of  use  of  the  service  is  also 
indicated  by  the  sum  of  the  amount  of  use  through  the  center 
and  other  sources.  Averages  and  totals  for  all  uses  of  the 
service  as  well  as  amount  of  use  of  the  service  obtained  from 
the  center  and  from  other  sources  are  calculated  using  the 
same  methods  as  those  given  above  for  amount  of  visits  and 
additional  uses  of  the  information  center. 

Both  importance  of  and  satisfaction  with  services  can 
generally  be  measured  and  we  recommend  that  these 
measures  be  done  as  shown  below.  In  this  way,  one  can 
compare  among  all  services;  for  example,  for  resource 
allocation.  However,  we  strongly  recommend  that  a  separate 
survey  be  done  for  some  services  using  a  critical  incident 
method  discussed  in  Chapter  9.  General  evaluation  of 
importance  and  satisfaction  can  be  measured  by  the 
following  survey  questi.^n: 

Q  We  are  intercsied  in  your  assessment  of  the  importance 

of  Information  Center  services  THAT  YOU  USE  for 
your  work  and  in  your  satisfaction  with  these  services. 

Please  rate  the  rmportance  of  and  your  satisfaction 
with  Information  Center  services.  NOTE:  RATINGS 
OF  IMPORTANCE  AND  SATISFACTION  (1  to  5) 
BELOW. 

Importance:  Not  at  all  important  —  1  to  Very 
Important  —  5 

Satisfaction;  Very  Dissatisfied  —  1  to  Very  Satisfied  — 
5 

Again,  one  can  expand  on  the  services  as  shown  above  for 
the  collection  of  materials.  Also,  it  may  be  useful  to  establish 
general  importance  and  satisfaction  with  specific  output 
attributes  of  services  such  as; 
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We  believe  that  importance  and  satisfaction  with  output 
attributes  of  some  services,  such  as  online  searches 
conducted  by  Center  staff  and  translation,  should  use 
critical  incidents  of  use  of  the  service  and  probably  should 
be  done  by  separate  survey  (see  Chapter  9). 

Finally,  the  General  Information  Center  survey  can  address 
availability  of  the  Center  (and  services),  competence  of 
Center  st^  and  specific  attributes  of  the  Center.  Examples 
of  importance  of  and  satisfaction  with  these  factors  aregiv»n 
below. 

Q  Please  rate  the  importance  of  and  your  satisfaction 
with  tile  Information  Center  facilities  and  staff.  NOTE: 
RATINGS  OF  IMPORTANCE  AND 

SATISFACTION  (1  to  5)  BELOW, 

Importance:  Not  at  all  important  —  1  to  Very 
Important  —  5 

Satisfaction:  Very  Dissatisfied  —  1  to  Very  Satisfied  — 
5 
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Importance  of  and  satisfaction  with  these  two  resources  (i.e., 
facilities  and  staff;  can  be  related  to  operational 
performance  measures  discussed  in  the  previous  chapter. 

Estimates  of  the  proportion  of  users  who  arc  satisfied 
(dissatisfied)  with  ser  ;es  or  who  consider  the  service  to  be 
important  to  their  work  are  discussed  below.  Also,  average 
ratings  ot  importance  and  satisfaction  can  be  calculated  as 
well.  Since  calculations  for  importance  and  satisfaction  are 
the  same,  only  one  example  is  given  below.  Also,  in  the 
example  the  scale  of  ratings  is  from  1  to  7  instead  of  1  to  5  as 
above. 


SMlsteciion  with  Ssrvtos 

Assponss 

Numbsr  of 

Rssponsss 

Proportion 
Rssponsss  04 

Vsry  DissstisRsd 

1 

2 

2  4% 

2 

5 

60 

3 

6 

71 

NsuvaJ 

4 

13 

155 

5 

19 

2^6 

6 

28 

333 

Vsry  Ssttsfied 

7 

11 

131 

TotsJ 

84 

1000 

In  this  example,  15.5  percent  of  the  respondents  arc 
dissatisfied  with  the  service  and  69.0  percent  are  satisfied 
with  it.  The  average  satisfaction  rating  can  be  calculated  by 
summing  responses  of  ratings  (422)  and  dividing  by  sample 
size  (84)  which  gives  5.02  average  satisfaction  rating 

Another  quick  method  of  calculating  the  average 
satisfaction  rating  is  to  cross  multiply  the  proportions  (not 
%)  limes  responses  and  sum  them  (i.e.,  0  024  x  1  +  0,060  X 
2  +  0.071  X  3  +  0.155  X  4  +  0.226  X  5  +  0.333  X  6  +  0.131 
X  7  —  5.022).  The  median  rating  is  4.94,  That  is  one-half  of 
the  responses  are  above  42.5  and  one-half  below  that 
number.  Thus,  the  median  is  between  4(26  responses)  and  5 
(45  responses)  and  42.5  is  0.94  of  45  (42.5  divided  by  45). 
The  median  rating  is  estimated  as  4  -I-  0.94  or  4.94. 

The  final  set  of  questions  deal  with  demographies  or 
characteristics  of  the  respondents.  There  are  two  general 
purposes  for  establishing  these  charactenstics: 

•  The  first  purpose  is  to  generally  characterize  the 
population  served  and  to  compare  sample  responses  to 
determine  if  the  responses  appear  to  be 
“representative”  of  the  overall  population.  This 
purpose  IS  useful,  but  not  as  important  as  the  second 
purpose, 

•  Charactenstics  of  persons  in  the  population  served 
should  help  explain  differences  in  use  of  information 
centers  and  their  services. 

For  example,  information  seeking  behavior  varies 
considerably,  by  work  role,  educational  background, 
location,  and  age.  Subdividing  the  population  into  groups 
that  reflect  use  is  referred  to  as  segmenting  the  market. 
Presently,  with  knowledge  of  usage  patterns  in  market 
segments,  information  center  managers  can  take  some 
corrective  action  to  increase  use  such  as  by  locating  small 
centers  in  underserved  sites,  publicizing  unfamiliar  services 
to  certain  segments,  improving  performance  of  services  to 
segments  that  are  more  discemable  with  service  attnbutes 
(e.g.,  medical,  legal,  scientific),  and  so  on. 

Examples  are  given  below  for  demographic  questions  we 
have  found  to  be  useful. 

Q  What  is  the  highest  degree  you  have  earned’ 


A»od*t*  I 

8«ch*loYs  (8A,  BS,  or  e<)ulv«}«nl}  2 

Matrer**  (MA,  M  S ,  M  BA  or  aquhraltnl} .  3 

Oorxoral*  (PHO,  ScO,  or  «<iutval«nt) .  4 

OjAr  (dwerKw) _ 

_ _ ...  5 
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Q.  In  what  year  did  you  receive  your  last/highest  degree? 
19 _ 

Q.  What  is  the  one  field  of  science  or  other  profession 
which  best  characleizes  your  area  of  work?  (If  more 
than  one  applies,  please  indicate  the  one  which  best 
characienzes  the  application  of  your  work). 


Sdtnw  «n<l  EiwIOMrinii 

Phyilcal  Scf#nc#  (Chemistry.  Physio:,  etc ) .  .....  i 

Msthemstics,  statistics .  .  2 

Computsf  Science .  ...  3 

Engineering .  4 


Other  Science  (specify) 

_ . .  5 

Bt/Hness 

Management .  6 

Business .  7 

Accounting .  .  ...  d 

Finarwe .  9 

Law .  ...  ...  10 

UberaiArts  .  .  ...  11 

Other  (specrfy) 

_ _  12 

0.  Which  category  best  dcs''nbes  the'work  role  in  which 
you  spend  the  largest  proportion  of  your  time? 


Management .  ....  .  .  1 

Research  &  Oevetopmenr  . .  2 

EducaOon/Tralning  ...  .3 

Operattoni .  4 

Administration  .  .  S 

Finance .  ...  $ 

Legal  ....  7 

Other  (describe)  . 

_  8 


Q.  What  is  your  current  location  (names  of  sites)? _ 

We  often  also  ask  respondents  to  indicate  any  special 
recognition  they  have  received.  For  example, 

Q.  Have  you  ever  leceivcd  any  awards  or  special 
recognition  at  X  Company? 


Yei  .  1 

No  .  2 

if  yes,  which  awards  or  special  recognition? 

Circle  al  that  apply 

Company's  Top  Award  .  a 

Named  Research  Felow  .  ...  b 

Director's  Award .  c 

Patent  Award  .  .  d 

Suggestion  Award .  e 

Other  (please  specify) 

_ _ _  f 


This  question  provides  an  indicator  of  the  value  of 
information  and  of  the  use  of  information  '■enter  services. 

Also  some  general  questions  concerning  infoimation- 
seeking  might  be  asked  in  the  demographic  section  of  the 
questionnaire.  For  example,  general  awareness  of  the 
information  center  and  journal  subscription  questions  might 
be  asked  here,  os  follows: 


Q.  Prior  to  this  survey  were  you  aware  that  the  X 
Company  has  libraries  for  general  use  by  employees? 

ym .  .  1 

No  .  ...  2 

Q.  How  many  personal  subscriptions  to  professional 
journals  do  you  receive?  (By  personal  subscription  we 
mean  one  which  is  personally  addressed  to  you  at  your 
home,  office  or  lab.) 

P*}d  fof  •ntif*ly  by  you  .  »ub*criptlons 

Pikf  iot  •ntir*fy  by  Company  . .  8ub»criptloo» 

Shared  by  you  and  someone  else  .  . .  ____  subscnptions 

Finally,  a  sensitive  question  deals  w-th  salary.  We  ask  this 
question  to  relate  user  time  to  cost  or  the  price  paid  for 
information  and  information  services.  Some  organizations 
object  to  this  question  and,  in  others,  the  question  is  optional 
as  follows: 

Q.  OPTIONAL 

What  range  represents  your  total  compensation  (i  e. 
company  salary  and/or  other  employment  income 


such  as  awards,  etc.)  in  the  last  year? 

L»$4  th*n  $25,000  .  1 

$25,000  -  $34,999  2 

$35,000  •  $44,999  .  3 

$45,000  •  $54,999 .  4 

$55.000 .  $54,999  ....  ....  5 

$65,000  or  ovtr  .  6 


The  method  for  calculating  average  salary  when  using 
ranges  is  the  same  as  that  described  above  for  distance  to 
information  centers.  If  an  organization  does  not  want  to 
include  salaries,  an  average  can  usually  be  found  from 
elsewhere  in  the  organization  (e.g.,  budget  or  personnel 
office).  However,  it  is  sometimes  useful  to  apply  the 
respondent's  salary  with  the  response  for  each  relevant 
question.  For  example,  suppose  we  want  to  place  a  cost  for 
going  to  the  information  center  in  order  to  determine  if 
“branches”  should  be  established  at  certain  sites.  One  knows 
how  much  time  of  each  respondent  is  currently  spent  going 
to  information  center  and  iternative  sources.  One  can  also 
estimate  how  much  time  would  be  spent  with  a  local  branch. 
Then  each  respondent  cost  can  be  calculated  by  the 
following  calculation  method: 

Estimated  Total  Cost  —  Estimated  annual  number  of  visits  x 
distance  to  the  center  (minutes)  x  60  x  salary  rate  per  hour. 

The  salary  rate  per  hour  is  found  by  dividing  total 
compensation  (plus  an  amount  for  fringe  benefits)  by 
number  of  hours  actually  worked  in  a  year  (c.g.,  2080  minus 
hours  of  sick  leave,  vacation  and  holidays).  In  the  US.,  a 
typical  hourly  rate  (including  fringe  benefits)  for 
professionals  is  $30.00  to  $38.00  per  hour. 

TTie  effects  of  market  segments  can  be  analyzed  by  merely 
cross-tabulating  importani  measures  with  responses  from 
the  segments  of  inteiest. 

8.4  Survey  orK.''own  Information  Center  Users 

There  are  basically  two  ways  to  administer  information 
center  user  surveys.  The  first  method  is  to  sample  a  list  of 
known  users,  for  example  when  there  are  registration  lists  or 
lists  of  clearinghouse  subsenbers  or  users.  Government 
agencies  are  more  likely  to  maintain  such  lists  than 
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companies.  Even  when  they  are  maintained,  such  lists  are 
often  outdated.  If  such  a  list  is  available  and  sample  forms 
available,  one  can  apply  .survey  questions  discussed 
previously.  A  second  method  for  internal  organization 
centers  is  to  sample  visitors  and/or  additional  users  of  the 
information  center,  ’’.'his  survey  provides  general  measures 
of  frequency  of  use  of  the  information  center  and  of  specific 
services,  importance  of  and  sahsfaction  with  access  to 
resources  (e.g.,  staff,  equipment,  collection,  etc.)  and 
services,  distance  of  users  to  the  information  center,  and 
user  characteristics.  This  survey  should  be  conducted 
periodically  over  time  (e.g.,  quarterly)  and  analyzed  once  a 
year  to  provide  best  results  and  to  minimize  seasonal  effects. 

The  visitor  survey  (sometimes  called  exit  survey)  is  relatively 
simple  to  administer.  Once  a  questionnaire  is  designed  and 
tested,  it  is  necessary  only  to  establish  a  sampling  scheme  by 
which  sampled  visitors  can  be  handed  questionnaires  to  be 
filled  out  in  the  center  or  mailed  (or  returned  at  a  later  visit) 
if  the  visitor  cannot  (or  will  not)  complete  it  on  site.  There 
are  two  difficulties  with  this  kind  of  user  survey. 

•  The  fust  difficulty  is  bemg  able  to  weight  or  project 
results  to  annual  totals  since  the  unit  sampled  is  visits 
(or  uses)  and  not  users.  Thus,  it  is  useful  for  the  center 
to  maintain  accurate  data  on  total  number  of  visits 
during  the  time  the  survey  is  being  conducted  (e.g.,  a 
week),  although  some  information  centers  maintain 
"gate  counts”  of  visits  over  a  year.  This  information  is 
imponant  for  estimating  annual  visits  and  number  of 
users. 

•  Second  since  the  unit  that  is  sampled  is  visits,  one  must 
distinguish  between  questions  dealing  with  visits  (e.g., 
number  of  times  a  card  catalog  is  used)  and  users  (e.g., 
user’s  satisfaction  with  aspects  of  the  collection,  user’s 
work  role,  etc.). 

It  is  more  difficult  to  estimate  both  number  of  visits  and 
number  of  users  through  visitor  surveys,  although  the  survey 
itself  is  generally  simpler  to  administer  and  is  less  expensive 
than  general  use  and  population  surveys.  Visitor  surveys 
must  be  stratified  by  number  of  visits  per  year  (or  month) 
because  sample  selection  is  based  on  visits  and  not  users. 
Thus,  frequent  visitors  (users)  have  a  greater  chance  of  being 
sampled  than  infrequent  visitors. 

As  described  previously,  number  of  visits  can  be  asked  on 
the  visitor  questionnaires  in  several  ways; 

Q.  How  many  times  have  you  visited  this  information 
center  in  the  past  year  (i.e.,  12  months)? 

_ times  last  year 


Q.  How  many  times  have  you  visited  this  information 
center  in  the  past  year  (i.e.,  12  months)?  (Please  circle 
as  appropriate). 

I  •  6  DtT)0$  ,  .  .  ...  ....  ) 

6  -  !0  . .  .  2 

.  3 

16  -  20  tkms .  4 

Mora  than  20  (apad^  how  many) 

flmat .  S 


Q.  How  often  do  you  (or  would  you  if  new  to  the  area)  visit 
this  information  center?  (Please  circle  one). 


La&s  than  onca  a  yaw .  i 

Onca  a  yaw  . .  2 

Savatal  Umaa  a  yaw .  3 

Onca  a  month  .  4 

Twica  a  month  .  ...  ...  6 

Onca  a  waak .  6 

Twtea  a  waak  or  mora  (apaofy  approidmataly 

how  many  timaa  a  waak)  fJmaaAiraak .  7 

To  estimate  total  and  average  visits  or  total  users  it  is 


necessary  to  post-slratify  by  number  of  visits.  An  example  is 
given  below  for  the  second  method  of  asking  number  of 
visits.  In  this  example  visits  are  sampled  over  a  one-week 
period.  A  total  of  301  visits  to  the  center  are  counted  during 
this  period.  A  total  of  152  visitors  answered  the  number  of 
visits  question  as  follows; 


No  OfVMtt 

No  of 
VUtore 

EttJmWOd 

Total 

No.  ofVlolts 

^Urnatad 

Ava.  No.  of 
VMS  per  Umt 

Eetimatad 

Total 

No  ofL'een 

1*5 

28 

2,664 

22 

1,311 

6>10 

3d 

4,017 

7.7 

522 

IMS 

32 

3,2!)6 

126 

258 

16-20 

11 

1,133 

176 

63 

Mof»  m*n  20 

42 

4,438 

365 

112 

TOTAL 

152 

15.766 

70 

2.266 

Of  the  152  visitors  sampled,  28  visitors  said  they  visited  1  to 
5  times,  39  said  they  visited  6  to  10  times,  etc. 


The  estimated  total  number  of  visits  is  found  by  muluplying 
the  number  of  visits  in  each  class  of  number  of  visits  (e  g.,  1 — 
5  visits  per  year)  by  a  weight  of  103  (i.e.,  52  weeks  times  301 
total  visits  per  week  divided  by  152  sampled  visits).  The 
weight  is  merely  the  estimated  total  number  of  visits  to  the 
center  in  a  year  (i.e.,  52  times  301)  divided  by  sampled  visits 
(152).  Thus,  the  estimated  total  number  of  visits  by  users 
who  visit  between  1  and  5  times  per  year  is  2,884  (i.e.,  28 
sampled  visits  times  a  weight  of  103).  The  estimated  total 
number  of  users  who  visited  the  information  center  this 
frequently  (i.e.,  between  1  and  5  times  per  year)  is  found  by 
dividing  the  estimated  total  number  of  visits  (e.g.,  2,884  total 
visits  for  those  who  visit  1—5  times  per  year)  by  the  average 
number  of  visits  (e.g.,  2.2  visits  per  person  per  year). 
Assuming  that  no  one  user  was  .sampled  more  than  once  in 
the  period  or  filled  out  more  than  one  questionnaire,  the 
average  in  the  1  —5  visits  class  can  be  calculated  in  the  same 
maimer  as  any  average  (i.e.,  the  sum  of  observations  divided 
by  the  sample  —  28  visits).  However,  when  responses  are 
given  as  a  range  one  must  estimate  that  average  in  another 
way.  If  the  number  of  visits  is  iog-normally  distributed  a 
good  way  >o  calculate  the  average  is  by  using  the  geometric 
average,  assuming  log-normality  is  reasonable. 

A  geometric  average  is  found  by  calculating  the  square  root 
of  the  product  of  the  range  point  (e.g.,  square  root  ot  1  times 
5  “  2.2,  square  root  of  6  times  10  -  7,7;  etc.).  Unless  one 
actually  has  values  specified  for  more  than  20  visits  (as 
shown  in  the  example  above),  the  average  number  of  visits 
for  those  who  visit  more  than  20  times  per  year  must  be 
calculated  by  extrapolation  of  the  log-normal  distribution  as 
shown  in  the  figure  on  the  next  page.  The  fact  that  the  plot  is 


CtmUative  Percentage 


nearly  straight  means  that  the  assumption  of  log-normality  is 
valid  (at  least  in  the  range  given).  Extrapolating  the  line  to 
the  98  percent  point  yields  a  value  of  about  78  visits  per  year 
(i.e.,  98  percent  of  the  users  visit  less  than  78  times  per  year). 
The  calculated  average  for  that  range  of  number  of  visits  is 
39.5  (i.e.,  square  root  of  20  times  78  ”  39.5). 


The  estimated  total  number  of  users  is  calculated  for  each 
class  of  frequency  of  visit.  Por  example,  the  total  number  of 
users  who  visit  between  one  and  five  times  per  year  is  1,311 
(i.e.,  2,884  total  visits  divided  by  2.2  visits  per  year).  Thus, 


the  estimated  total  number  of  users  across  all  classes  of 
frequency  of  visits  is  2,26(i. 

A  less  formal  way  of  asking  number  of  visits  was  given  above 
with  responses  such  as  “less  than  once  a  year,”  “o.nce  a  year,” 
“several  times  a  year,”  etc.  Most  of  these  responses  imply  an 
average  rather  than  a  range.  Thus,  once  a  year  would  be 
counted  as  1 ;  once  a  month  —  12;  twice  a  month  —  24;  etc. 
Several  times  a  year  would  have  an  estimated  average  of  4.6 
(i.e.,  square  root  of  2  times  11  assuming  log-normality). 
Estimated  total  visits  and  total  user  are  calculated  in  much 
the  same  manner  as  above. 
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Estimating  tlie  total  number  of  users  for  each  class  of 
frequency  of  visits  (i.c.,  stratum)  is  essential  because  one 
must  use  these  numbers  to  weight  the  responses  to  each 
question  that  is  relevant  to  users.  Some  questions  on  the 
visitor  survey  questionnaire  will  be  relevant  to  visits  and 
some  will  be  relevant  to  users.  For  example,  t^ical 
questions  related  to  visits  and  users  are  as  follows; 

Visit-Related  Questions 

•  What  did  you  do  on  your  visit  to  the  information  center 
today? 

•  Did  you  ask  the  staff  for  help  finding  tlie  information 
and/or  matenals  you  were  looking  for? 

•  For  what  purpose  did  you  need  the  information/ 
matenals  sought  in  the  information  center? 

•  Approximately  when  did  you  arrive  at  (or  leave)  the 
information  center  today? 

User-Related  Questions 

•  How  satisfied  are  you  with  the  ability  of  the 
information  center  staff  to  help  you  locate  needed 
information  and  materials? 

•  How  satisfied  are  you  with  the  information  center’s 
collection  of  books  and  other  materials 

•  What  is  your  primary  work  role? 

Some  questions  can  be  relevant  to  either  visit  or  user 
depending  on  how  the  question  was  asked.  For  example; 

•  "How  long  does  it  normally  take  you  to  get  to  the 
information  center?”  is  user-related. 

•  "How  long  did  it  take  you  to  get  to  the  information 
center  on  this  visit?”  is  visit-related. 

As  indicated  above,  calculating  measures  from  user-related 
questions  is  not  straight-forward  when  made  ffom  a  visitor 
survey.  To  demonstrate  this,  an  example  is  given  below  for 
estimating  the  proportion  of  users  who  work  in  research  and 


development  (R&D)  and  other  work  roles.  In  this  example 
there  were  200  visitors  who  responded  to  the  question 
concerning  their  work  lole  (R&D  and  other). 

Without  weighting  one  would  assume  from  the  sample  that 
48.0  percent  of  the  users  are  primarily  engaged  in  R&D  (i.e., 
73  divided  by  152  —  0.480).  However,  weighting  by  number 
of  users  in  each  stratum  yields  a  much  different  result. 
Estimates  in  each  stratum  are  calculated  by  taking  he 
proportion  of  R&D  and  other  professionals  in  each  stratum 
and  multiplying  them  times  the  number  of  users  in  the 
stratum.  For  example,  the  proportion  of  R&D  professionals 
in  the  stratum  of  those  who  visit  one  to  five  times  per  year  is 
estimated  to  be  0.321  (i.e.,  9  divided  by  28).  Thus,  the 
number  in  that  stratum  is  363  (i.e.,  0.321  times  1,311)  and 
the  number  of  other  professionals  is  948  (i.e.,  0.679  times 
1,311).  Considering  all  classes  of  number  of  visits,  it  is  found 
that  the  weighted  estimate  of  proportion  of  users  who  are 
R&O  professionals  is  32.9  percent  (i.e.,  746  divided  by 
2,266)  compared  with  the  48.0  percent  calculated  from 
unweighted  data.  Actually,  48.0  percent  is  the  proportion  of 
visits  that  are  by  R&D  professionals.  In  this  example  the 
weighted  estimate  is  so  different  from  unweighted  results 
because  R&D  professionals  tend  to  be  frequent  visitors  (c.g., 
18%  visited  more  than  15  times),  whereas  other 
professionals  tend  to  visit  mfrequently  (e.g.,  39  visited  more 
than  15  times)  Yet,  the  sample  was  even  more  pronounced 
(55%  vs  16%). 

The  average  number  of  visits  by  R&D  and  other 
professionals  are  10.0  per  person  for  R&D  professionals 
and  5.5  visits  per  person  for  other  professionals.  These 
averages  are  calculated  by  multiplying  average  number  of 
visits  in  each  straram  times  the  totail  persons  (e.g.,  2.2  times 
363),  summing  over  the  five  strata  and  dividing  by  the  total 
number  in  the  population  cf  interest  (e.g.,  746  R&D 
professionals).  Note  that  the  unweighted  estimates  would  be 
23.0  and  12.0  visits  per  person  respectively,  for  R&D  and 
other  professionals.  Thus,  proper  weighting  substantially 
affects  estimates  of  proportions  and  averages,  as  well  as 
totals,  thus  avoiding  biased  or  inaccurate  estimates. 


EXAMPLE  OF  ESTIMATES  CONCERNING  VISITORS 
FROM  VISITOR  SURVEY 


R&D 

OTHBH 

BOTH 

NUMBER 

OF 

verrs  sampi£ 

BAMPlf 

PROP. 

(%) 

POP. 

7X3TAL 

SAMPLE 
SAMPLE  PROP. 

(%) 

POP. 

TOTAL 

POP. 

SAMPLE  TOTAL 

1-5 

9 

32.1% 

363 

19 

67.9% 

948 

28 

1,311 

6-10 

11 

28.2 

147 

28 

71.8 

37 

39 

522 

11-15 

13 

40.6 

105 

19 

59.4 

153 

32 

258 

16-20 

8 

72.7 

46 

3 

27.3 

17 

11 

83 

More  than  20 

32 

76.2 

85 

10 

23.8 

27 

42 

112 

TOTAL 

73 

48.0% 

746 

79 

52.0% 

1,520 

152 

2,266 

Chapter  9 

Evaluating  the  Effectiveness  of  Speciflc  Information  Center  Services 


9.1  Background 

In  this  chapter  we  discuss  evaluation  of  several  specific 
information  center  servtces  from  the  perspective  of  users. 
This  evaluation  is  different  from  the  evaluation  presented  in 
Chapter  8  in  that  the  evaluation  in  this  chapter  delves  much 
more  in-depth  concerning  satisfaction  with  service  output 
attnbutes  such  as  quality  and  timeliiiess,  purpose  of  use  of 
the  service,  and  consequences  of  use  of  the  service.  Again,  a 
user  survey  is  the  principal  method  used  and  a  critical 
incident  of  a  recent  use  of  the  service  it  the  basis  for  much  of 
the  analysis.  In  particular  we  focus  on  the  following  services; 

•  online  bibliographic  and  numeric  database  searching, 

•  Current  Awareness  Bulletins,  and 

•  journal  routing. 

The  basic  philosophy  of  measures  that  we  have  discussed  in 
other  chapters  holds  here  and  would  apply  to  other  services 
as  well  It  is  noted  that  performance  of  the  operational 
aspects  of  the  services  above  (i.e.,  input  costs,  output 
quantities  and  attributes,  productivity,  etc.)  are  covered  in 
chapters  found  in  Part  2.  Measures  of  the  value  of  these 
services  and  models  relating  linkage  of  output  attnbutes  to 
amount  of  use  and  consequence  of  use  arc  covered  in 
chapters  in  Part  4. 

The  sampling  frame  for  surveys  of  specific  services  usually 
consists  of  lists  of  recent  users  of  the  services. This  is  true  for 
in-house  information  centers  and  independent  centers  such 
as  clearinghouses.  We  say  recent  because  the  survey  will  ask 
about  a  recent  use  of  the  service  and  some  services  such  as 
online  searching  and  access  to  article  copies  through 
document  delivering  services  are  relatively  infrequent.  Thus, 
recall  or  memory  of  the  details  of  this  last  use  will  begin  to 
dim  after  about  three  months.  If  tliere  is  no  list  of  users,  one 
can  select  the  sample  from  a  list  of  recent  uses  and  apply  the 
method  described  in  Chapter  8  for  estimating  number  of 
uses  from  samples  of  visitors.  This  is  not  a  difficulty  because 
most  questions  arc  use-related  rather  than  user-related 
anyway.  In  fact,  if  the  sample  is  of  users  it  may  be  necessary 
to  weight  by  uses  as  desenbed  m  detail  in  the  next  chapter 
for  document  reading. 

9.2  Online  Bibliographic  and  Numeric  Database 
Searching 

In  this  survey  we  obtain  data  and  information  about  the 
extent  of  online  bibliographic  and  numeric  database 
searching  obtained  from  the  information  center  and  a  series 
of  questions  concerns  a  critical  incident  of  searching  done 
by  the  center.  The  survey  begins  by  acquainting  the  user  with 
the  fact  that  they  have  recently  used  the  service.  In  our 
examples  below  we  focus  on  online  searching  of  external 
literature  databases.  A  statement  such  as  the  following  might 
be  useful. 


You  have  been  identified  as  a  recent  user  of  the 
Information  Center  Online  Searching  Services.  These 
services  include  searching  external  literature  databases 
(e.g.,  Dialog,  BRS,  ORBIT,  CASOnline,  etc.)  online, 
and  other  sources  for  identifying  and  verifying  needed 
documents.  The  questions  below  refer  to  your  use  of 
this  service. 

0  How  many  times  have  you  or  someone  on  your  behalf 
used  the  Information  Center  for  brief  references  or 
full-searches  in  the  last  year  (i.e.,  12  months)? 

No.  ol  briol  No.  ol  Ml 
lOlManctt  warchos 
lastyur  last  yew 

a  Used  th©  service  rnysell  ...  --- 

b  Asked  iXtleaQue/subordmate  to 

use  the  service  for  me  .  - _ 

Note  that  we  do  not  ask  about  use  of  alternative  sources  such 
as  searching  themselves,  using  a  subordinate  to  do  the 
searchmg  themselves  or  using  an  external  source  (e.g.,  an 
academic  library  or  broker).  This  information  should  be 
obtained  from  the  General  Population  Survey  in  order  to  get 
unbiased  results  estimated  from  nonusers  as  well  as  users. 
Also,  we  ask  respondents  to  indicate  number  of  searches  for 
a  full  year  since  there  are  usually  relatively  few  uses  of  this 
service  by  one  person  over  a  year.  Typically,  more 
professionals  use  the  service  themselves  rather  than  asking  a 
colleague  or  subordinate  to  do  so.  Finally,  bnef  reference 
searches  tend  to  outnumber  full  searches  by  about  two  to 
one  (say,  five  or  six  to  two  or  three). 

We  then  focus  on  a  critical  incident  or  the  most  recent 
search.  Usually  we  concentrate  on  full  searches,  although  it 
is  not  necessary. 

The  questions  below  refer  to  your  most  recent  use  of 
Information  Center  Online  Searching  Services 
involving  the  last  IN-DEPTH  FULL  SEARCH  done 
for  you. 

0  What  was  the  general  topic  for  which  this  last  in-depth 
full  search  was  done  for  you? 

Topic: _ 


This  question  is  largely  asked  to  get  i  cspoiidents  to  focus  on 
a  specific  search  done  for  them. 

0  On  this  last  literature  search,  approximately  how  many 
documents  or  citations  were  identified,  what 
proportion  did  you  obtain,  and  what  proportion  did 
you  (or  do  you  intend)  to  read? 
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NumtMT  Pnpoilioo  Proportion 

IdtnIIIM  Obtointd  Rood 

iyp«  otllMi  (%)  (%) 


a.  Journal  artidas  _  _ _  _ 

B.  Booka  _  _  _ 

c.  Extamai  tacBnical  raports _  _  _ 

d.  Patent  documanta  _  _  _ 

a.  Confaranca  procaadinga  _  _ _  _ 

t.  Other  (spacity)  _  _  _ 

Averaged  across  searches,  we  find  that  journal  articles  tend 
to  dominate  responses  with  external  technical  reports 
following  in  number.  Usually  fewer  than  one-half  of  the 
citations  are  read. 

The  next  set  of  questions  deal  with  the  performance  and 
effectiveness  of  the  service.  It  can  be  in  a  section  labeled  as 
such. 

0  Did  you  discuss  or  negotiate  the  time  it  would  take  to 
perform  the  search  with  the  searcher? 

Yes  . 1 

No . 2 

If  yes,  what  was  that  duration  of  tum-around  time? 

_ Hours  or _ Days  or _ Weeks 

Surpnsingly  to  us,  we  find  that  most  information  centers  do 
not  discuss  or  negotiate  turn-  around  time  with  users,  or  at 
least  users  are  not  conscious  that  they  do.  Of  course,  in  data 
analysis  this  time  and  the  one  below  must  be  converted  to  a 
common  time  unit  for  analysis.  Days  seems  to  be  the  best 
unit. 

Q  From  the  time  you  initially  contacted  the  Information 
Center  to  perform  the  in-depth  full-search,  how  long 
did  it  take  to  recc.ve  the  search  results? 

_ Hours  or _ Days  or _ Weeks 

This  result  should  be  compared  with  the  negotiated  time 
when  appropriate. 

Q  How  important  was  the  response  time  for  this  last 
search? 

Cirde  one 

Not  (I  an  Important .  1 

Moderately  Important .  2 

Important  .  3 

Vary  Important . 4 

Extremely  Important  .  5 

Q  From  the  standpoint  of  response  time  of  the  search, 
were  you: 

Cirdeone 


Very  diaaalnited  .  I 

Weeatlalied  .  2 

Nelthef  aatialled  not  diasatlalled  .  3 

Salialled .  4 

"<7  aatbned .  5 


Analysis  of  the  response  time  (or  difference  in  negotiated 
and  response  time),  importance  of  response  time  and 
satisfaction  with  this  output  attribute  provides  a  good 
picture  concerning  performance  regarding  this  service 
attribute.  Further  discussion  relating  this  and  other 
attributes  to  extent  of  service  use  is  given  in  Chapter  11. 

Q  To  your  best  knowledge,  was  there  any  relevant 
information  missed  in  this  last  search? 

yea  .  1 

No  .  2 

Donl  Know  ...  3 

Most  of  the  time  users  indicate  “no”  or  “don’t  know”. 
However,  this  question  can  be  correlated  with  importance  of 
and  satisfaction  with  relevance  of  search  output.  These  two 
questions  are  asked  as  follows; 

Q  How  important  is  the  relevance  of  information  to  your 
expressed  needs  for  this  last  search? 

drde  ono 

Not  tt  til  Importamt  .  1 

Moderatoly  importtot .  2 

Important .  3 

Voiy  important .  4 

ExUamely  important .  .  . .  •  5 

0  From  the  standpoint  of  the  relevance  of  information  to 
your  expressed  needs,  were  you: 

Cirefo  one 

Very  dIesaUsfied .  ...  .1 

Dissatisfled .  ...  2 

Neither  <atl$fied  nor  dteeabsfied .  3 

Setialled  .  4 

Very  satisfied  .  5 

The  next  questions  deal  with  number  of  items  or  citations 
identified  on  the  search.  The  actual  number  (given  in  a 
previous  question)  can  be  related  to  importance  of  and 
satisfaction  with  this  service  attribute. 

Q  How  satisfied  were  you  with  the  amount  of  information 
presented  to  you  in  the  search  output? 

Cirde  one 

Very  dissatisfied .  1 

D'^^atisfied  ...  2 

er  satisned  nor  dissatisfied  3 

Satisfy .  4 

Very  satisfied ...  5 

Q  If  dissatisfied,  did  you  get  too  much  or  too  little? 

Cirde  onn 

Too  much .  .  ....  1 

Too  little .  . .  2 

Typically,  users  get  too  little  rather  than  too  much,  although 
in  one  organization  the  reverse  was  frequently  true  leading 
to  a  new  policy. 
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Q  How  important  is  the  amount  of  information  presented 

to  you  in  the  search  output? 

Circte  on« 


Not  «t  aH  impoftant  . .  1 

Modifstaly  in^^orUnt .  2 

Important  .  3 

Vary  important .  4 

Extramaly  important . 5 


Some  organizations  charge  for  search  service.  However, 
even  when  they  do  charge  users  may  not  be  aware  of  the 
amount  due  to  charging  policy.  Nevertheless  questions 
concerning  user  charges  can  be  useful  for  evaluation. 

Q  Did  you  pay  for  this  last  in-dcpth  full-search? 

Qrda  otm 


Yaa .  t 

K  yaa.  how  much .  . 

No .  2 

Ooni  Know .  3 


Q  From  the  standpoint  of  the  usefulness  and  value  of  the 
search,  was  the  price: 

Cirefa  or>o 

Much  too  htgh .  .  t 

Too  high .  2 

About  right .  .  3 

Too  low  .  4 

Much  too  low  for  valua  racalvad  .  $ 

Q  How  important  was  the  price  in  your  decision  to  use 
the  search  service? 

^de  one 

Not  at  all  importv)t  ,  .  .....  f 

Moderately  Important  .  2 

Important .  ....  3 

Very  Important  ....  4 

0ctremely  important  .  ...  5 

Finally  m  this  section  one  might  want  to  ask  some  questions 
about  searcher  competency  from  the  perspective  of  users. 

This  perspective  is  useful  to  compare  with  known 

competency  and  to  correct  if  the  user’s  perception  is 
generally  not  valid.  Note  that  another  format  for  the 
question  is  given  as  an  example. 

Q  From  your  perspective,  how  important  is  the  searcher’s 
skill  in  conducting  the  online  research?  Circle  the 
appropnatc  rating. 


Not  at  al 
important 

Important 

Very  important 

1 

2 

3 

4 

Q  From  your  perspective,  how  well  qualified  was  the 
searcher  concerning  knowledge  of  the  subject  field 
researched?  Circle  the  appropnatc  rating. 

Not  am  OualHad  Veiyqi««ed 

quatUad 

12  3  4  5 


Q  From  your  perspective,  how  important  is  the  searcher’s 

knowledge  of  the  subject  field  searched?  Circle  the 
appropriate  rating. 


NotataV 

Important 

Important 

VeryimportM 

, 

2 

3 

4 

1 

O  How  satisfied  are  you  with  the  researcher’s  skill  in 


searching? 

Ctidaona 

Vaiy  dissatiafled  . 1 

DiwatitMd .  2 

Naittiar  satollad  nor  diasatlallad .  3 

SaSatlad .  4 

VMy  satlallad .  5 


Q  How  satisfied  are  you  with  the  searcher's  attitude? 

drdaona 

Vary  dlasadstled . t 

Disutisllad .  a 

Nalthar  salialted  nor  dasalistied  . .  3 

Satlillad .  4 

Vary  satiatiad . . . . . .  ....  S 


Q  Please  feel  free  to  comment  on  any  aspects  of  the 
searcher's  knowledge,  skills,  or  attitude  that  you  feel 
are  important. 


Q  Please  feel  free  to  comment  on  any  other  aspect  of  the 
Information  Center’s  Online  Searching  Service. 


The  final  set  of  question’s  deal  with  the  purposes  and 
consequences  of  the  Online  Searching  Service.  These 
question’s  might  be  incorporated  into  a  special  section  of  the 
questionnaire  These  question’s  relate  to  (D.  E,  F  and  f  on 
the  framework  Figure  4  in  Chapter  2). 

Q  Please  indicate  the  primary  activity  for  which  you  used 
the  results  of  this  last  search. 


Acllvitv  Cifcle  only  oi» 

Resoarch~ba$lc  or  product  development  .  ...  i 

Engineering-research  or  manufacturing/processing  etc  . .  .  2 

Technical  or  research  support  (e  q ,  QC,  standards, 

consumer  research,  etc)  .  3 

Manageme’*:  or  executive  .  4 

Administrative .  5 

Fir>ance  or  accounting  6 

tagal/Petent  7 

Medical .  8 

Operotions  (e  q .  facilities  engineering,  mail  serv.  etc )  9 

Background  research . 10 

’’rofessiortal  development .  . .  1 1 

Other  (please  soectM  . .  12 


* 
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Q  Please  indicate  for  which  communications  purposes 
this  last  starch  was  primarily  used. 

CIrcIt  only  ono 


Not  don*  lor  communic«t>on  purpooM .  1 

MormU  ditcuMlon  witti  cotKoguos .  2 

Conoultfng  or  orMtlng  olMrt . . .  3 

Mtidng  fomwl  Inttmol  pr«MnUtlon«  .  * 

Making  formal  axtamal  pratantatlana .  S 


WWIng  (raporti,  propoaala,  artidaa,  ale.)  .  a 

Othar  (plaata  tpacify) _ r 

The  pre-coded  responses  should  correspond  exactly  to  any 
other  surveys  that  obtain  information  about  primary  work 
activities  or  communication  activities  (see  Chapter  10).  In 
this  way  results  can  be  correlated  or  cross-checked.  These 
questions  particularly  contribute  to  (D)  on  the  framework  in 
Figure  4. 

Contribution  to  the  professionals’  work  output  attributes  (F 
in  the  framework  figure)  is  determined  from  the  document 
reading  survey  (see  Chapter  10).  In  that  survey,  respondents 
are  asked  about  readings  of  documents  that  are  identified  by 
online  searching;  including  the  consequences  of  reading  on 
quality,  timeliness,  etc.  of  the  professionals’  work.  If  that 
survey  is  not  done  (or  responses  are  deemed  too  sparse  to 
reveal  th:  consequences  of  searching)  then  the  user  output 
attribute  questions  can  be  incorporated  in  this  survey. 

A  series  of  questions  may  also  be  asked  to  determine  one 
perspective  of  the  value  of  the  Online  Search  Service. 

Q  In  what  ways  did  you  benefit  by  having  a  staff  member 
from  the  Information  Center  perform  this  in-depth 
full-search  for  you? 

CMS  tpplcabto 


I  stvtd  llms  lor  myssif/sulf .  1 

If  so.  spproxlmststy  flow  many  mlnutss  or  hours? 

_ mlnutss  or _ hours 

Thsy  could  psilorm  ths  sssrch  fsstsr  than  I  could .  2 

Thsy  could  psrform  ths  sssrch  bsttsr  than  I  could .  3 

Olhsr  (pissjs  spsdiy) _ 


Q  If  Information  Center  could  not  have  provided  you 
with  the  in-depth  full-search,  how  would  you  probably 
have  done  the  search? 

drds  any  Ifist  you  would  hsvs  used 


I  would  not  have  dons  ths  search  I 

I  would  have  done  the  search  myself  onlna  ....  2 

I  would  have  dons  ths  search  myself  manually .  3 

I  would  ha. a  delegated  ths  search  to  someone 

siss  on  my  stall .  4 

I  would  have  used  another,  external  Irbraiy .  5 

I  would  have  catsd  a  ccSsagus  who  Is 
knowlsdgssbis  In  ths  subject  field .  6 

I  would  have  used  a  coniractor/outsids  senfes .  7 

Other  (specify) _ _ 

_  8 


Q  How  much  time  was  spent  by  you  or  someone  from 
your  staff  in  obtaining  the  in-depth  full-search? 
Approximately  how  much  MORE  would  it  have  cost 
you  to  get  it  from  the  alternative  source(s)  you  selected 


in  the  question  above.  If  you  cannot  make  a  reasonable 
guess,  please  enter  "DK”  (Don’t  Know). 


Actual  AdcWonal 
Urns  finis 

Your  cwn  tims  (discussing  sssrch,  sic ) .  . ^niln.  min. 

Tims  cf  soTiisons  alas  cn  your  staff  (If 

you  did  or  would  ask  thsm  for  assistancs . . ^min.  _ min 

Pass  assodatsd  with  search  services  (If  applicable)  .  3 _  3  . 

Telephone  calls .  3 _  3 _ 

Other  costs  (please  specify) _ 

_ 3 _  3 _ 


Tire  first  question  provides  an  indication  of  the  value  of 
having  the  Online  Search  Service  perform  the  search  (f  on 
the  framework  Figure  4). 

The  second  question  gives  evidence  of  information  that 
would  be  lost  if  there  was  no  Online  Search  Service 
(typically  about  20-30%  of  searches  would  not  be  done)  and 
what  alternate  sources  might  be  used  (usually  they  would 
search  themselves  either  manually  or  online). 

The  third  question  (1st  column)  indicates  the  amount  of 
time  that  users  spend  with  discussing  the  search,  etc.  and  in 
other  related  costs.  This  indicates  part  of  the  “price”  paid  by 
users  to  use  the  Online  Search  Service.  The  question  also 
determines  how  much  more  it  would  cost  users  to  use 
alternative  sources  (2nd  column).  We  consider  this 
additional  cost  to  be  the  most  important  indicator  of  the 
value  of  this  service  (see  Chapter  13).  Typically  this  value  is 
two  to  four  times  the  actual  cost  of  the  service. 


9.3  Current  Awareness  Bulletin 

Many  organizations  prepare  (or  purchase)  Current 
Awareness  Bulletins  in  order  to  keep  users  up-to-date  with 
the  current  literature.  In  a  sense  these  bulletins  complement 
or  replace  purchase  of  personal  subscnptions  and/or 
journal  routing.  This  service  is  quite  expensive  and, 
therefore,  worthwhile  evaluating  from  time  to  time  to  ensure 
that  it  is  performing  satisfactorily.  The  information  center 
usually  has  a  list  of  users  and,  therefore,  a  sampling  frame  is 
easily  obtained.  However,  the  General  Population  Survey 
may  reveal  that  the  bullehns  are  passed  on  to  persons 
beyond  the  user  list.  If  so  a  second  stratum  should  be  formed 
to  complement  the  list  of  known  users. 

There  are  usually  several  bulletins  published  and  distributed 
by  the  information  center.  Two  questions  help  determine 
extent  of  u.se, 

Q  Do  you  currently  receive  a  Current  Awareness  Bulletin 
issued  by  ’he  Information  Center? 


Yes  . 1 

No . 2 


Q  Which  Current  Awareness  Bulletins  do  you  receive 
and  approximately  how  often  do  you  normally  refer  to 
a  specific  issue?  Note  that  you  may  sometimes  refer  to 
a  bulletin  that  you  do  not  receive. 
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CM*  V  Awig*  no. 
youraoilvtd  odltnMyou 
ttitBuMtn  i*lwto«i* 
Mu*  BuMjn 

B<o«:i*nc*i  ■  Olnteal  PfoOuel*  BulMln .  >  _ 

Cn*mlc*l  BulMUn .  2  _ 

Cottmg  Un*  ol  Ttchnology 

•  PlukiMcIlon  .  3  _ 

*  V*cuum/M*gn*tlc«/Optle*l  1*01100 .  *  _ 

UOrwy  BultUn* .  *  _ 

T*ctmlc*l  R«port  BulMtln .  *  _ 

Note  that  use  is  for  a  single  issue.  Thus,  total  use  must  be 
projected  to  all  issues  (c.g.,  six)  and  from  sample  responses 
to  number  in  the  population  surveyed. 

The  remaining  questions  deal  with  a  critical  incident  of 
the  most  recent  use  of  a  bulletin. 

Q  What  is  the  last  Current  Awareness  Bulletin  that  you 
referred  to? 

CM*onl)ron* 

Blo*d*nc*«  ■  Clinical  Product*  Bullalin .  1 


Cti*mlc«l  Bullatin .  S 

Coatinfl  Un*  ol  Ttctmology 

•  Fluid  tacUon .  a 

*  Vacuum/M*fln*tic*/Optlc*l  s*ction .  4 

Ubrary  Buflatm* .  S 

Tachnic*!  Ropert  BuMui .  6 


This  question  helps  focus  on  the  most  recent  use  and  also 
provides  a  cross-tabulation  for  evaluating  specific  bulletins. 

Q  Approximately  how  many  issues  back  was  that  last 
Bulletin  to  which  you  referred? 


Mo«t  r*c«nt  on«  . 1 

TWO  .  2 

ThfOf .  3 

OWor . 4 

Don't  know  . 5 

Q  For  what  reason  did  you  refer  to  this  last  Bulletin? 

Browiin^  moft  r«C4nt  istu« .  1 

n4m*>7«bfrfd  •  rff«r«nc«  I  n«4d4d .  2 

Othtr  fiooctM  _ _ 

_ ...  3 


These  two  questions  help  determine  whether  not  the 
bulletins  are  used  for  retrospective  searching  (say,  in  lieu  of 
online  searching).  There  is  little  evidence  from  the  studies 
we  have  done  that  bulletins  are  used  in  this  way. 

Q  Approximately  how  much  time  did  you  spend  reading 
or  using  this  last  Current  Awareness  Bulletin? 

_ Minutes 

This  question  provides  some  indication  of  the  “price”  paid 
to  use  the  bulletin  and,  therefore,  is  an  indicator  of  it’s  value 

Several  questions  correspond  to  the  questions  used  for  the 
Online  Search  Service  in  Section  9.2.  Such  questions 
include: 

•  data  on  number  of  items  identified  and  read. 

•  purposesofuse, 


•  importance  of  the  bulletin,  and 

•  comments  about  the  service. 

Other  similar  questions  that  are  worded  differently  than  the 
Online  Search  Service  are  as  follows: 

0  In  what  ways  did  you  benefit  by  having  the  last  Current 
Awareness  Bulletin? 

Ctreto  al  i«)pK»bto 

I  Mv*d  tinw  tor  mytM  or  ttiiff  In  ktontifying 

oMdod  document*  .  1 

*  It  to.  approximotoly  how  m«ny  hourt?  Hour* 
ktontiftod  n**d*d  *ourcM  toonor  than  I 

could  hava  othtrwi** .  2 

IdanMad  naadad  aourca*  that  I  probably 

would  not  hava  idantiflad  otharwita .  3 

Olhar 


Q  If  the  Current  Awareness  Bulletin  was  not  provided  to 
you,  how  would  you  have  identified  the  needed 
sources?  Select  only  the  method  you  would  most  likely 
use. 

Circta  orUy  on* 


I  would  not  hava  dona  It .  t 

Dapaitmant  drculatlon/routlng .  2 

I  would  hava  conductad  an  onllna  aaarch .  3 

I  would  hava  dalagated  an  onllna  saarch  to 

tomaona  ei*a  on  my  staff .  4 

Othar  tsoadM 

_ .  .  5 


Q  Approximately  how  much  MORE  would  it  cost  you  to 


use  this  other  source? 

Of  my  own  Uma .  . .  t>our* 

Tima  of  aomaona  aba  .  . hours 

To  purchasa  tha  saarch  sarvica .  >_ 

Talaphona  can* .  . S  _ 

Othar  tsoaeffy) 

_  $ _ 


O  Please  rate  your  general  level  of  satisfaction  with  the 
following  attributes  of  the  Current  Awareness  Bulletin 
you  last  used? 

^Mlhar 

Saliaflad 

Vary  Nor  Vary 

OissatMad  OissatMad  Oissatistlad  Satbflad  Satafod 


1  2  3  4  5 

Satisfaction  Rating 
(1-5) 

Currancy  Of  itams/anula;  kfamiffad 

In  Cu'rant  Awaranas*  Bullatin  ....  ... 

Cevaragt-narthar  too  much  nor  loo  liMa . 

Quality  of  raproducbon .  _ 

Format  of  Information  In  Currant  Awaranas*  Buflatin  . . 

Ilam*/antr1aa  gfvan  without  an  abstract  . 

Easa  of  ordaring  Currant  Awaranas*  Builaffn .  __ 

Daisy  of  racalpt .  _____ 

For  the  last  question  it  is  useful  to  have  respondents 
elaborate  on  attributes  which  are  found  to  be  unsatisfactory. 
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9.4  Journal  Routing 

Wc  have  found  journal  routing  to  work  very  well  in  some 
organizations  and  very  poorly  in  others.  A  survey  of  journal 
routing  can  be  done  as  part  of  a  document  reading  survey 
(see  Chapter  10)  or  independently.  A  survey  concerning  this 
service  involves  known  users  and  possibly  some  users  who 
receive  copies  from  persons  on  the  routed  lists.  Below  arc 
some  questions  that  address  evaluation  of  this  service. 
Detailed  results  of  the  consequences  of  this  service  applies 
to  questions  from  the  document  reading  survey. 

Q  Do  you  receive  circulated/routed  journals? 


Yes  . 1 

•  How  many  journals  (not  issues)? _ 

No . 2 


Perhaps  not  too  surprisingly,  occasionally  persons  indicate 
they  do  not  receive  this  service  even  though  they  are  on  a 
routing  list.  If  the  proportion  who  indicate  “no"  is  high,  some 
follow-up  is  warranted. 

At  this  point  a  set  of  critical  incident  questions  should  be 
addressed  concerning  the  respondent's  last  reading  of  a 
routed  journal. 

Q  What  position/number  are  you  on  this  journal 
circulatton/routing  list?  (e.g.,  1st,  2nd,  etc.) 

Position/number _ 

Q  Approximately  how  long  after  the  journal  circulation/ 
routing  began  did  you  receive  it? 

_ Days  or _ Weeks  or _ Months 

Q  From  the  standpoint  of  lime  for  the  journal  issue  to  get 
to  you,  were  you: 

Ctrds  one 


Very  OiuatisHed  .  1 

WsMlIsllee .  2 

Neither  seSefied  nor  dissatielied . 3 

Setlslled .  4 

Very  setleSed .  5 


These  questions  together  provide  strong  evidence 
concerning  the  appropriate  nunjber  of  persons  who  should 
be  on  a  journal  routing  list.  We  find  that  six  to  eight  persons 
is  about  the  right  number.  Also,  these  questions  help  to 
identify  trouble  spots. 


0  Approximately  how  many  articles/itcms  did  you  read 
from  this  jouma.  issue  and  how  many  aniclcs  did  you 
or  someone  on  your  behalf  photocopy? 

_ arlicle.s/ilems  read 

_ artieles/items  photocopied 

Tliis  question  provides  the  basis  for  estimating  the  total 
number  of  articles  read  as  a  result  of  using  this  service.  By 
comparing  cost  per  article  read  with  other  sources  of  journal 
articles  (e.g.,  personal  subscriptions,  office  collections, 
information  center  collection)  we  typically  find  that  cost  per 
reading  is  substantially  less  for  this  service.  However,  the 
length  of  routing  lists  must  be  kept  short  or  satisfaction  falls 
off  rapidly. 

On  occasion  we  have  examined  the  possibility  of  alternatives 
to  current  journal  routing  procedures.  An  example  of  a 
question  concerning  alternatives  is  given  below. 

0  There  are  some  alternatives  to  journal  routing  or 
possible  changes  that  could  be  made  in  the  service. 
Please  rank  your  preferences  for  five  possible  services 
from  1  (highest  rank)  to  5  (lowest  rank). 

Rtnk 

a.  Kasp  journal  cJrcutatioiVroiiting  unchanpad . .  .  _ 

b  Kaap  journal  drculatiorVroutlna  llata  ahon  (by 
irKraaaIng  numbar  of  coplaa  purchaaad) .  __ 

c  Kaap  journal  drculaiioiViouting  Hats  abon  by 
raaMcdng  routing  lisi  accaaa  to  paopla  witn  valid 
roasona  for  racaMng  the  particular  journal .  .  _ 

d.  Eliminate  journal  drculation/roullng  altogatbar, 
but  maintain  currency  by  keeping  all  new  journal 
Issues  In  a  dedicated  place  in  a  nearby  Informatron 
Center  Site  so  lliey  would  be  available  to  an  users  _ 


a  Replace  Journal  drculatlorVroutlng  witn  immediate 
maffing  to  Individuals  (electronic  or  paper)  of  the 
TableB  of  Contents  of  new  journal  issues  Tbese 
current  issues  vrould  be  available  In  a  dedicated 
place  in  a  nearby  Informrtlon  Center  Site  . ,  , . 


Note  that  this  question  ranks  alternatives,  as  opposed  to 
rating  them.  The  ranks  can  be  averaged  or  displayed  for  each 
alternative,  keeping  journal  circulation/routing  lists  short 
(by  increasing  number  of  copies  purchased)  is  usually 
ranked  highest. 


Chapter  10 

Evaluating  the  Higher  Order  Effects  of  Information  &  Information  Service  Use 


lO.t  Background 

We  refer  back  to  the  framework  given  in  Chapter  2,  Figure  4 
and  Figure  5.  In  particular,  this  chapter  addresses  (D),  (E), 
(F)  and  (f)  measures  given  in  the  framework  in  Figure  4.  A 
survey  is  recommended  that  establishes  time  spent 
performing  various  activities  (e.g.,  research,  legal  work, 
managing,  finance,  marketing,  etc.).  The  survey  also 
determines  extent  of  communication  input  (e.g.,  quantities 
and  time  spent  reading,  listening,  etc.),  and  output  {e.g., 
quantities  and  time  spent  writing,  advising  others,  making 
presentations,  etc.)  related  to  the  work  of  professionals  in  an 
organization.  Specific  survey(s)  are  recommended  for 
document  reading  regarding  such  information  as  depth  of 
reading,  how  readers  identified  documents  read,  where  they 
got  the  documents,  and  consequences  of  reading.  Since 
information  centers  provide  identification  and  access  to 
many  of  the  documents  read,  one  can  determine  the  extent 
to  which  centers  contribute  to  the  use,  usefulness  and  value 
of  information  found  in  documents  (i.e.,  D,  E,  F  and  f  in  the 
framework). 

10.2  Information  Input  and  Output 
In  order  to  determine  amount  of  time  spent  by  professionals 
doing  their  primary  work  and  communicating  one  can  ask 
three  questions  dealing  with  (1)  total  amount  of  time  spent 
working  in  a  year,  (2)  proportion  of  this  time  involved  in 
vanous  primary  work  activities  (work  roles),  and  (3) 
proportion  of  their  time  involved  in  communicating  by 
various  means.  Examples  of  these  three  questions  are  given 
below.  The  first  question  deals  with  amount  of  time  spent  in 
work-related  activities.  Each  organization  has  an  accepted 
or  scheduled  amount  of  time.  For  example,  in  the  U.S.  a 
typical  work  year  for  profe-ssionals  involves  2,080  hours 
with  about  1 20  hours  of  vacation  time,  88  hours  of  holidays 
and  an  average  of  about  52  hours  of  sick  leave  or  1,820  total 
hours.  However,  professionals  often  spend  additional  hours 
on  work-related  activities  such  as  working  late,  reading 
while  travelling  to  and  from  work,  attending  evening 
professional  meetings,  and  so  on.  It  is  difficult  and,  perhaps 
not  meaningful,  to  try  to  distinguish  between  regular  hours 
and  other  time  spent.  Thus  one  can  establish  an 
organization’s  normal  annual  work  hours  (e.g.,  1.820  hours) 
and  add  to  that  additional  lime  by  asking  the  following 
question. 

Q  Approximately  how  many  hours  per  year  do  you 
devote  to  ORGANIZATION  work  or  your  own 
professional  development  that  are  in  add'uion  to  the 
normal  eight-hour  work  day?  Include  such  time  as 
working  late  (or  early),  reading  while  travelling  to  and 
from  work,  attending  professional  meetings  during 
non-working  hours,  etc. 

Ctrda  approprixlo  code 

None .  I 

MOO .  2 

101-250  .  3 

251-500  « 

501-1,000  .  5 

Over  1,000  (tpecityj  _ houre .  S 


We  find  that  average  amount  of  such  "over-time"  ranges 
from  150—300  hours  among  organizations.  A  typical 
organization  result  is  as  follows; 


Proportion 
on  Raspoiuee 


None .  251% 

1-100 .  133 

101-250  .  71  8 

251-500  .  18  4 

501-1,000  .  13.1 

Over  1,000  (everege  1,234  hours)  .........  83 


For  this  company,  average  lime  is  296  hours  per 
professional  per  year  in  addition  to  1 ,820  for  a  total  of  2,116 
average  total  hours  per  professional  per  year.  Average  salary 
plus  fringe  benefits  for  this  company  is  $52,500.  Therefore, 
average  hourly  rate  is  $24.81  ($52,500  divided  by  2,1 16).  To 
that  we  often  add  about  50  percent  for  overhead  (i.e..- 
$37.22). 


The  second  question  deals  with  amount  of  time  spent 
working  on  specific  primary  activities.  This  can  be  done  by 
asking  for  the  proportion  of  time  spent  (multiplied  by  total 
time  spent  determined  from  the  question  above).  This 
question  is  asked  as  follows: 


Q  Please  indicate  the  general  kinds  of  activities  that  you 
perform  in  your  work  and  the  approximate  proportion 
of  time  you  spend  over  a  year  doing  it,  including  lime 
spent  working  for  The  Organization  outside  of  normal 
working  hours.  Do  not  include  vacation,  holidays,  sick 
leave,  etc.  in  calculating  proportions 


NBvtr  pwfonn 

nimwyActMijr  tctMty 

(ptoo*  tMdD) 

Oi  pfotfyet  d«v*topm*nt  A 

CnjpoMfing-fMBafcn  Of  maoufBCtu  tjf 
pfOc*»»in9,  MC  B 

TKhniCBl  Of  rmveh  tiippcrt  (•  9 ,  qu»*V  contro),  JUndord* 
cofi«um«f  iHovctf.  ot; )  C 

MAnBgomont  or  0 

MmrMtiion  C 

f^nvic*  Of  MOomMing  P 

G 

H 

OpofBtfom  (•  9 .  crsgin*»rin0,  mai  sorvtcM.  »tc }  I 

BKkgrounj  imifcfi  J 

Pro<w$Jen«i  ()«vt>opfntnt  K 

09m  (p»tw  tpociN)  L 


Al  propoftiom  onUfod  mug(  toM  100% 


Pfoporttofi 

spent 

W 

% 


100% 


Since  some  types  of  professionals  tend  to  work  more  lime 
than  others,  one  can  establish  estimated  proportion  of  lime 
for  each  activity  by  multiplying  each  proportion  reported  by 
total  time  (e.g,  1,820  -F  geonietnc  means  for  reported 
amount)  for  each  respondent,  then  add  times  across 
respondents  for  each  activity  to  determine  the  total  time  for 
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that  activity.  The  overall  proportion  for  each  activity  is  then 
a  straightforward  calculation. 


reading)  is  obtained  in  another  questionnaire  (Section  10.3 
of  this  chapter). 


Overall  time  spent  communicating  by  various  means  is 
determined  on  the  next  question.  In  this  question  one  can 
ask  professionals  to  indicate  the  relative  time  they  spend 
actually  doing  p.imary  activities  and  how  much  time  they 
spend  communicating  in  various  ways. 

Q  Above  you  indicated  the  primary  activities  in  which 
you  arc  engaged  in  your  work.  Now  we  would  like  to 
know  how  you  spend  your  time  actually  doing  these 
activities  In  particular  wc  arc  concerned  about  the 
communication  component  of  your  work. 

Please  indicate  the  proportion  of  your  time  performing 
the  activities  above  in  actually  thinking,  conducting 
experiments,  and  so  on  and  the  time  spent 
communicating  in  various  ways. 


Ntfver  Perform  Proportioo 

ActfvNy  thk  wOMtf  of  tim« 

(ptMse  d^)  spent 
(%) 


ActusKy  doing  the  thinking,  analysis, 

•xperimanls,  accounting, ate .  A 

informal  dtscusslona  wrth  coUeaguas  .  8 

Consulting  or  advWng  others  .  C 

Making  formal  interrud  presentations .  0  _ 

Making  formal  external  presentations  .  .  E 

AtterKfing  formal  mtemal  presentations  ...  F  . 

Attending  formal  extemsl  presentations  ...  Q 

Writing  (reports,  proposals,  ar^ea.  etc )  . .  H 

Reeding  (reports,  proposeis.  books, 

artides.  etc )  l 

Other  (ptesse  soeerM 

_ ...  d  _ 

Al  propodkxts  must  total  100%  100% 


Referring  to  (he  schema  depicting  communication  in 
organizations  we  have  identified  the  amount  of  time  devoted 
to  work  input  \nvo\v\n%: 

Proportioo  of 
total  time  (%} 

e  Interpersonal  convnunlcation 

from  internal  sources .  20% 

from  external  sources .  1 


•  Communicatron  by  documents 

from  Internal  docoments .  3 

from  external  documents  ....  9 


e  interpersonal  communication 


for  internal  redplenta .  10 

for  external  redplarrts .  * 


Quantities  of  information  output  are  observed  from  the 
following  set  of  questions.  The  first  question  deals  with 
amount  of  time  spent  consulting  or  giving  substantive 
advice. 

Q  Approximately  how  many  individual  times  did  you 
consult  or  give  subsiantivc  advice  to  others  m  the  past 
month  (30  days)? 

_ limes  in  the  past  month 

In  this  question  we  suggest  using  a  montli  for  the  reporting 
lime  because  we  have  found  that  those  who  say  they  consult 
or  give  substantive  advice  (typically  70%  of  professionals), 
do  so  rather  frequently  (typically  15—20  times  per  month  or 
200  limes  per  year),  'fhus,  wc  find  that  about  180  hours  are 
spent  giving  substantive  advice. 

One  can  also  partially  measure  output  by  the  number  of 
formal  records  written  basically  for  internal  use. 

Q  In  the  past  year,  how  many  formal  records  of  your  work 
(c.g.,-  technical  reports,  laboratory  notebooks,  legal 
bnefs,  software  programs,  etc.)  have  you  wnllcn  or 
sub.stantially  contributed  to? 

Sols  Author  CoAuthor  Condbulor 
Tschnlcal  Rspons  __  ___  ___ 

Lsboratoiy  Notsbooks  ____  _ 

Lsgsl  btisU  ___  ___  ___ 

Othsr  rspons  _  _  _ 

Sollwars  programs  ___  __  __ 

Other  (pisase  spoeV)  _  ___  _ 


Here  it  is  useful  to  establish  the  terminology  used  in  the 
organization  for  reports.  Some  organizations  have  very 
formal  definitions  for  different  kinds  of  reports  (e.g., 
technical  memos,  technical  briefs,  technical  reports,  etc.)  An 
example  of  responses  m  an  organization  is  given  below. 


Proponkxi  Avg.  no  psr  Avg  no 

professlonais  professtonals  psr  aX 

who  writs  (%)  who  writs  prolsssiooals 

Tschnlcal  rspons  _  _  2  3 

laboratory 

notsbooks  _  _  1 .0 

Isgal  brisfs  _  _  8  9 

Othsr  rspons  _  9  8 

Softwa/s  programs  _  _  4  2 

Other  _  7.8 


As  a  guide  we  have  found  that  reports  average  about  three 
co-authors  and  two  other  contributors.  Thus,  total  output 
quantities  arc  adjusted  by  these  factors.  For  example,  if  a 
person  indicates  they  are  a  sole  author  of  two  rcpoits  and 
co-author  of  four,  they  are  said  to  have  wntten  3.33  reports. 


•  COfh.'W.'hfcatio.nc  by  rfoewnsnU 


lor  fntsmal  rsciplsnlt .  7 

for  sxtsrnsl  rsolpisnts  .  83 


Some  quantities  of  information  input  are  obtained  in  the 
questions  below.  Amount  of  reading  (and  time  spent 


We  assume  that  books,  formal  articles  and  often  similar 
publications  are  written  largely  for  external  consumption, 
although  not  entirely  so.  The  extent  to  which  these 
documents  are  written  depends  a  great  deal  on  the  type  of 
organization  involved  and  the  organization’s  policies.  Few 
articles  and  books,  for  example,  are  written  by  company 
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authors  Yet  as  shown  above  they  spend  an  appreciable 
amount  of  time  writing.  Professionals  at  the  National 
Institutes  of  Health  in  the  U.S.  spend  even  more  time  writing 
and  almost  exclusively  for  external  consumption.  An 
example  of  the  question  to  obtain  this  information  is  given 
below. 

0  In  the  past  year  how  many  external  publications  have 
you  authored,  co-authored  or  contributed  to? 

Solt  Author  Co-Author  Corttnbutor 

Accepted  manusalpts  o*. 
schoiady  )ournid 
artides 

Accepted  and  comptoted 

book  nwinuscnpts  ____ 

Other  publicationa  (e  g . 
conferertce  proceadinga. 
chapters  In  books,  etc ) 

Journal  articles  average  four  to  six  co*authors.  Books  and 
other  publications  only  average  about  1.8  co-authors 

One  can  obtain  data  about  the  number  of  internal 
presentations  given  as  well  as  the  time  making  presentations 
and  attending  presentations  m  the  next  question 

Q  In  the  past  year,  approximately  how  many  formal 
presentations  have  you  made  (including  shared 
presentations)  concerning  any  of  your  work  activities, 
average  length  of  the  meetings  (in  hours),  and 
approximately  how  many  total  people  attended  all  of 
these  specific  kinds  of  meetings? 

Type  o<  Meeting  Number  of  Avg  length  of  Total  r>o  of 
meetings  each  meeting  anendees  at 
afl  meetirtgs 

a  Informal  worker's 

meetings  ____  _  - . 

b  Formal  internaJ 

meetings  _  _  _ 

c  Division  meetings  _  _  — ^ 

d  Other  _  .  _  - 

In  the  example,  one  company  reported  averaging  9.9  formal 
internal  meeting  presentations,  lasting  an  average  of  1.9 
hours  with  an  average  of  7.8  persons  attending  the  meeting. 
This  comes  to  an  average  of  18.8  hours  making  the 
prc.scntalions  and  147  hours  attending  this  kind  of  meeting 
In  a  sense,  the  amount  of  time  spent  attending  the  meetings 
IS  a  “first  order”  indicator  of  the  value  of  this  information 
output.  Because  professionals’  time  is  a  scarce  resource, 
their  choosing  to  spend  this  lime  listening  to  the 
presentations  is  an  indicator  of  the  “price”  (and,  therefore 
value)  they  are  willing  to  pay  for  the  information.  External 
presentations  can  be  observed  in  much  the  same  manner. 

Q  In  the  past  year,  approximately  how  many  workshops, 
seminars,  university  classes/courses  have  you  (or  you 
with  others)  conducted  or  co-presented?  (Count  all 
individual  cla.sses  of  university  courses.) 

Total  Tow  Tow  numbw 

numbtr  houra  of  attanUaaa 

Intamal  _  _  _ 

Extamal  _  _  _ 

Unlvaiaily/collagt  _  _  _ 


Finally,  the  number  of  written  proposals  and  plans  are 
obtained. 

0  In  the  past  year,  approximately  how  many  written 
proposals  or  plans  did  you  prepare,  how  many  have 
been  accepted  and  how  many  are  still  pending? 

Nunbar  ot  propoaali,  plana  pispaiad  .  . . 

Numbar  partially  accaptad  .  . .  ...  — 

Numbar  antiraly  accaptad  ...  — 

Numbar  rajactod . — 

Numbar  panding .  — 

One  can  ask  about  acccptar.ee  and  rejection  to  obtain  an 
indicator  of  the  quality  of  the  work. 

The  communication  injiut  resources  and  costs  and  output 
quantities  are  summarized  for  a  typical  organization  in 
Tables  10.1  and  10.2. 

10.3  Surveys  of  Document  Reading  am'  Use 

We  suggest  that  data  be  collected  from  one  or  more 
document  reading  surveys.  Since  most  reading  involves 
journals  (  scholarly  and  trade),  books  (business,  scholarly, 
reference,  etc.)  and  technical  reports  (internal  and  external) 
three  separate  document  reading  surveys  may  be  warranted, 
although  they  can  be  combined  into  a  single  survey  as  is 
given  in  the  examples  below.  These  surveys  should  be  from  a 
population  of  users  and  potential  users  of  an  information 
center  (c.g.,  all  professionals  in  an  organization).  The 
examples  below  rely  heavily  on  a  critical  incident  of  reading 
and,  therefore,  the  survey  is  a  pseudo  two  stage  sample  That 
IS,  the  first  stage  sampling  unit  is  people  (e.g .  professionals) 
and  the  second  stage  sampling  unit  is  a  specific  reading, 
although  usually  only  one  such  reading  is  observed  Some 
questions  involve  the  person  sampled  (e.g..  amount  of 
reading,  amount  oflibrary  use,  and  demographics), but  most 
of  the  questions  involve  the  reading  (e.g.,  type  of  document 
read,  how  it  was  identified,  where  was  it  obtained,  and 
consequences  of  the  reading).  Each  type  of  question  involves 
a  different  kind  of  estimation  process  which  we  will  describe 
below. 

The  sample  design  required  to  .sample  individuals  (i.e.,  1st 
stage)  should  be  essentially  the  same  as  described  in  the 
General  User  Survey  (Chapter  8).  Also  most  relevant 
information  behavior  and  demographic  questions  discussed 
there  apply  in  this  kind  of  survey  as  well.  It  is  useful,  for 
example,  to  ask  about  number  of  visits  and  additional  use.s 
and  similar  questions  on  all  surveys  of  the  general 
population  because  they  can  be  combined  in  a  single 
database  to  increase  the  overall  sample  size  for  these  highly 
relevant  que.stions. 

In  order  to  weight  or  project  the  cntical  incident  reading  to 
the  population  total  of  readings  for  a  year,  it  is  necessary  to 
obtain  estimates  of  total  readings  by  type  of  document  read 
Later  we  will  show  how  these  numbers  should  be  used.  Since 
amount  of  reading  is  relatively  frequent,  we  suggest  using 
one  month  as  a  time  period  for  observation.  Just  as  with 
information  center  visits  and  other  measures,  one  can  ask 
this  question  in  several  ways  One  suggested  way  is  as 
follows: 

Q  In  the  past  month,  approximately  how  many  of  each  of 
the  following  types  of  documents  have  you  read  in 
connection  with  your  work  at  The  Organization. 
Reading  is  defined  as  going  beyond  the  title,  contents 
page,  and  abstract  of  the  document. 
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TABLE  10.1 

INFORMATION  INPUT/RECEIVED 


1VP*  ol  Mornution 


_ WmH  CommunKaDow 

Hours  OuartUH 


Exlamaf  Communlcatlom 
Hom  Quantitits 


Recorded  ItVormatton 

Reading  books  _ 

Reading  journals  ^ 

Reacting  externa!  reports  or  patent  documents  0 

Reading  internal  reports  and  memos  39 


61  19  books 

6S  101  artictes 


0  Unknown 

70  reports  and  memos  ^ 


21  reports 
or  patent  docs 


Total  39 

Interoorsortal  Inforrrtation 

Professional  development  Unknown 

Formal  presentations  128 

Informal  discusskxts  73 

Receiving  consuttatiorVadvice  225 


Total  426 


94  readings  126  145  readings 

Unknown 

49  nieetings  attended 
Unknovm 
Unknown 


49  30  10 


30  10  meetings  attended 


TOTAL 


TABLE  10.2 

INFORMATION  OUTPUT/SENT 


Internal  Commurwcalions 

External  CorrYnunicalions 

Type  o(  Mornution 

Hours 

Quanuies 

Hours 

Ouarttios 

Recorded  Information 

Wntir^  reports 

87 

Unknown 

Unknown 

Technical  reports/protocols 

33  repons/protocols 

Unknown 

Unknown 

Technicai  memoranda 

6.6  technical  memoranda 

Unknown 

Unknown 

Standards/spectfications 

5  5  st^ards/spoc 

Unknown 

Unknown 

Original  data  (eg.  tab  notes,  etc) 

18.0  original  data 

Unknown 

Unknown 

Regt^ar  memoranda 

469  memoranda 

Unknown 

Unknown 

Writing  proposais/plans 

67 

5  0  proposals/plans 

Unknown 

Unknown 

Writing  books 

— 

- 

None  reported 

None  reported 

Writing  articles 

— 

— 

2 

006  articies 

Wntlng  other  pubAcatlons 

— 

4 

0.1 1  other  pub 

Writing  patent  applications 

- 

- 

Unknown 

0  08  patent  apphc. 

Total 

154 

853 

6 

0  25 

imeroersonal  Information 

Consukin^yQMng  advice  to  others 

225 

258  occasions 

Unknown 

Unknown 

Educatmgn’ralnlng  others 

59 

1.6  $es8>on5  conducted 

Formal  presentations 

2Z5  meetings  conducted 

1  3  sess  cond 

Informal  discussions 

7  5mtg  conducted 

73 

Unknown 

Unknown 

Unknown 

Total 

403 

282 

90 

68 

TOTAL 

557 

96 

1 
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No.CiRMdnoi^ 

Um  In  P»  1  Month 

Professional  jourr^s  or  law  reviews 

___  Trade  journals,  buBetins,  non'techn{c&  magazines,  newr>ienors.  etc 
___  Professional,  legal  technical  or  bushess  booK«> 

Reference  books,  handbooks,  directs  »n«'s.  manuals  etc. 

Standards  and  specification^ 

_  Proprietary  information  (technical  reports,  analytical  methods, 
competitor  anaVsis) 

Laboratory  notebooks  (internal  to  tho  Con^any) 

_  External  reports  (a  g ,  govern'nent  documents,  (.'Slertts,  etc) 

Respondents  cannot  recall  exactly  how  mai*y  documciiis 
they  have  read,  etc.,  but  they  have  a  go-jd  gineral  idea.  We 
have  found  our  estimates  to  be  close  to  estimdtcs  achieved 
using  other,  perhaps  more  accurate,  methods.  It  is  specified 
that  respondents  provide  approximate  answers  because  they 
will  feel  frustrated  if  they  think  exact  values  arc  required. 
Since  approximations  are  adequate  one  can  also  ask  (h,. 
questions  using  ranges  of  number  of  readings  .such  as: 


No  of  f— dktpiANW  In  pMt  month 

TyinoldoouMnliMd  Non.  1.2  M  MO  O*  to  (tpoc*,) - 


Calculations  of  average  amount  of  reading  can  be  done  bv 
using  mid-points  or  geometric  avera'cs.  For  example,  for 
one  company  results  are  as  follows; 


NO.O( 

RMdingi 

Avvrag« 

fto  0( 
fMtponm 

Proportion 

W 

0 

0 

131 

\ 

1 

34 

».  i 

2 

2 

31 

i.i 

3-5 

39 

42 

15 

6-10 

78 

21 

78 

Ov«r  10 

20.9 

17 

6.2 

276 

1000 

By  cross-multiplying  number  of  responses  times  estimated 
average  (0X131-HX344-2x31-l-3.9X42-l-7.8X21-f2O.9X 
17)  divid^  by  276  one  gets  aiv  ,stimated  average  of  2.81 
readings  per  month  or  33.7  readings  per  year.  One  can  make 
more  refined  estimates  for  projecting  one  month  to  a  year  as 
described  in  Chapter  8. 

A  series  of  questions  concerning  the  critical  incident  of 
reading  can  then  be  asked.  These  questions  help  determine 
the  proportion  of  readings  that  came  from  the  information 
center,  how  the  documents  were  identified,  how  thoroughly 
the  documents  were  read  and  the  consequences  of  reading. 
For  each  example  question  below  we  will  indicate  how  the 
measure  fits  into  the  evaluation  framework  discussed  in 
Chapter  2. 

In  the  questionnaire  there  should  be  a  section  devoted  to  the 
critical  incidents  of  reading.  The  section  should  have 


instructions  that  make  it  e'ear  that  answers  refer  to  the 
critical  incident.  For  example: 

SECTION  2 

DOCUMENT  READING 

All  questions  in  this  section  refer  to  the  document  that  you 
read  most  recently  (related  to  your  wo>-k  or  professional 
development).  Please  note  that  it  does  not  matter  how  long 
ago  tliis  last  document  was  read. 

Q  What  type  of  document  did  you  last  read  for  work- 
related  purposes? 


Pfol*f«lon*l  lounuM  a  law  inlaws .  1 

TiaUa  loumala,  boMlina.  non-lachnical 

maga^a,  nawatattan.  ate  .  2 

I’lofatalonal,  lagal.  tactmlcal  or 

butlnata  books  .  3 

fialafanoa  books,  handbooks. 

dliacloilat,  manuals,  alo. .  4 

Standanlt  and  apadtioaliona .  s 

Propilataiy  inlorrnalion  (Mchnical  rapoilt, 

ankVtlcal  malhodt,  compatllor  analyilt/  .  S 

Uboratory  noiabooka 

(Intainal  to  tha  Company) .  7 

Bdarnal  lapoit:  {»  g.,  govammant  docunanta, 
patanta,  ate ) . . . .  .  ...  -  a 


We  mention  “work-related  purposes”  because  we  have 
found  that  respondents  sometimes  report  non-work-relaied 
reading  (particularly  booksy  established  from  the  next 
question.  All  the  remaining  respr-nses  are  cross-tabulated 
and  analyzed  by  type  of  document;  sometimes  into  such 
groups  as  journals,  books  and  technical  reports. 

Q  What  was  the  title  or  topic  of  this  last  tead”documcnt? 
If  the  document  was  a  journ'd  article,  refer  to  thearticle 
—  not  the  journal  title.  Approximate  document  title  or 
topic: _ 


This  question  is  n'-ed  primarily  to  make  sure  that  the 
respondents  focus  on  h  s;.Pnfic  -ncident.  However,  we  have 
also  found  that  general  topics  ihcinselves  have  been  useful 
for  analysis  in  some  instances. 

Q  If  this  last  document  read  was  a  journal  a,  tide, 
approximately  how  many  articles  did  you  read  from  the 
journal  from  which  the  article  was  read  in  the  last  year 
(12  months)?  (SKIP  IF  THE  LAST  READ 
DOCUMENT  WAS  WTAN  ARTICLE) 

_ articles 

This  question  is  used  to  analyze  professionals’  economic 
trade-off  concerning  use  of  personal  subscriptions,  journals 
in  office  v  'iections  and  information  center-provided 
journals.  Depending  on  distance  to  the  office  collections  and 
information  centers,  we  find  tliat  professionals  tend  to  use 
personal  subscriptior.s  when  they  read  more  than  about  10 
articles,  per  year.  This  question  is  also  used  to  estimate  the 
total  number  of  journals  read  by  professionals,  which  is 
typically  about  13  journals  ir  which  at  least  one  ariicle  is 
read. 


94 


J  large  number  of  professionals  personally  subscribe  to 
.■jumals  in  which  they  read  fewer  than  10  articles  we 
>-ccoinF.^nd  to  them  that  they  reconsider  this  decision  and 
point  oiit  the  economic  benefit  of  using  an  information 
center  t  s  obtain  irrirequcntly  used  journals. 

Q  In  wha?  year  was  this  document  published  (c.g.,  article 
written,  etc.) .  19 _ 

Tlii'  question  provides  an  indication  of  the  age  distiibution 
of  documents  read.  Also,  we  find  that  most  journal  articles 
over  two  svars  ead,  for  example,  are  read  from  journals 
provided  b>  infoiiiiation  centers.  We  generally  recommend 
that  individuals  and  office  collections  discard  journal  issues 
over  two  years  ole  unless  used  very  frequently. 

Q  Approximately  how  many  previous  times/o-casions 
have  you  read  i,r  referred  to  this  document? 

_ '<mes/occa«ions 

We  correlate  this  response  with  method  of  identifying  a 
journal  article,  book  or  report  (s:e  question  below  on  how 
the  respondent  found  out  'ibnut  the  document). 

1\vo  questions  give  one  an  indication  of  how  thoroughly 
documents  are  read. 

Q  How  thoroughly  did  you  read  this  document? 

OcM  on*  004* 


wtm  g/**t  c*n  .  i 

Whh  •ttwitlon  10  th*  m*ln  point* .  2 

JuM  to  g«<  Itw  W** .  3 


0  What  is  your  best  estimate  of  the  lime  in  hours  or 
minutes  that  you  spent  reading  the  document  this  most 
recent  lime? 

_ Hours  or _ Minutes 

The  time  data  are  used  (along  with  number  of  readings)  to 
estimate  the  total  time  professionals  spend  reading.  This  is 
an  indicator  of  the  value  they  place  on  the  information  read. 
Also,  we  find  that  documents  (particularly  journal  articles) 
provided  by  information  centers  are  read  in  more  depth  and 
for  a  longer  period  of  time  than  documents  obtained  from 
other  sources.  This  result  is  not  too  surprising  since  most 
readings  of  journal  articles  read  from  personal  subscriptions 
are  recently  published  articles  (typically  80%  of  the  readings 
from  articles  published  less  than  six  months  ago).  These 
articles  are  not  often  read  in  depth,  but  rather  mostly  for 
current  awareness. 

These  data  are  used  to  estimate  amount  of  information  used 
by  professionals  (D  in  Figure  4),  amount  of  input  resources 
(time)  used  to  perform  an  activity  ^£  in  Figure  4),  and 
proportion  of  the  readings  from  documents  obtained  from 
information  centers  (f  in  Figure  4). 

The  next  question  deals  with  how  professionals  identified 
the  document  read. 

Q  How  did  you  initially  find  out  about  this  last  document 
you  read? 


Old*  ma*t  •ppropflM*  ood* 

» lounwl  ontv 


Fouvl  wNi*  browsing  tlw  joume'  Issut: 

•  ol  •  clrculU*<t/rout*d  Jounul .  1 

•  ol «  p«it«n»l  subscription .  2 

•  d  on  olllc*  coltscllon  JoumsI . .  3 

•  ol  sn  inlornuillon  c*ntsr  joumsl .  4 


Found  wbit*  browsing  lb*  shsivss  of  th* 

Informotlon  conttr .  S 

Found  whil*  browsing  th*  c*lslogs  ol  th* 

Inlormstlon  contsi .  6 

Roccmnwnrlsd  by  Inloimsiton  ctnltr  tisll  .  7 

Al.dBwmtnii 

From  snoth*r  porton  (*  q ,  *  cotlsagu*, 

•ulhot,  *rc.) .  e 

CItod  In  snollwt  publication  (I  *.,  anld*, 

book.  *10. .  9 

Citod  in  Ih*  output  ol  *  compulsilzsd 

SiiMitut*  a**rch  .  10 

Cllod  In  s  print  copy  ol  •  Cmoni  Awsr*n«s» 

Bulatin  .  It 

CItod  In  snothM  pilM*d  Indsx .  12 

Olhor  (ploss*  sptdfy) _ 

_ ■  13 


This  question  (together  with  the  one  below)  indicates  the 
extent  to  which  specific  information  center  services 
contnbute  to  reading  (D,  E,  and  f  on  Figure  4).  The  specific 
se’~dccs  include,  for  example,  access  to  the  collection, 
computer  literature  searches.  Current  Awareness  Bulletin, 
journal  routing,  catalog,  ccntrali2cd  purchasing  of  books, 
etc. 

Q  From  which  source  did  you  get  this  last  document  that 
you  read?  Note  that  purchase  of  journal  subscrip.ions 
include  those  journals  obtained  through  professional 
society  membership. 

Ckcl*  most  appropriats  cod* 


A  |oum«J  subscription  or  book  U>at  I  paid  for .  1 

A  {oumal  subscription,  book  or  technical 
report  that  the  Organization  paid  for  (not 

lnformattor>  center)  .  . ,  .  2 

A  Journal  subscription,  book  or  technIcaJ 
report  that  the  informabon  censer  purchased 

for  me  to  Keep  (I  e ,  centralized  ordenng) .  3 

An  information  center  copy  ordered  upon 

my  request .  4 

A  copy  of  a  Journal  article  obtained  by  the 
information  center  upon  my  request  (from 

intertibrary  loan  or  document  deliver/  service)  .  5 

A  copy  routed/circulated  by  the 

Information  center . . .  6 

A  copy  routed/drculeted  by  someone  else .  7 

An  office  collectioo .  8 

An  external  library  (academic,  pubiic.f to)  * .  9 

A  colleague  or  co-worker  .  10 

The  author  (not  employed  by  the  Organization) .  1 1 

Other  (pleaae  aoectfv) 

_ ..  12 


Topically  about  25  to  35  percent  of  articles,  25  to  50  percent 
of  books  and  30  to  60  percent  of  teclmical  reports  are 
obtained  from  information  centers. 
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The  next  set  of  questions  deals  with  the  purposes  and 
consequences  of  reading  (D,E  and  F  on  Figure  -t),  and  i  te 
contribution  that  the  information  center  makes  to  reading 
(D,  E,  F  and  f).  The  first  two  questions  follow  the  qu.-stion 
concerning  source  of  the  document  read.  Ihese  two 
questions  together  are  used  to  Jetcrmine  one  perspective  of 
value  of  the  information  center. 

Q  If  you  could  not  use  the  source  .specified  above,  where 
would  you  have  obtained  tfie  document  or  equally 
useful  information? 

Cird*  on*  cod* 


Would  not  h*v*  obttintd  tha  dooumani 

Of  intofmallon  .  > 

From  •  conatgua .  2 

From  a  conauitant .  a 

From  anotlw  library  (plaaaa 

apaoHy) _ •  .  * 

From  my  own  coSactlon .  5 

I  would  hava  bought  it . a 

Omar  (plaaa*  apaeffy) _ 


Q  How  much  time  did  you  spend  locating  and  acquiring 
the  document?  If  you  spent  time,  but  don’t  know  how 
much,  indicate  "DK."  Enter  your  response  in  the  First 
eolunji.  If  you  had  to  go  to  an  alternative  source  for  the 
document  or  information,  approximately  how  much 
MORE  time  (than  you  actually  spent)  would  you  need 
to  identify  and  acquire  the  document  or  information? 
Enter  your  response  in  the  second  column. 


ACTUAL  am  (ipani 

AOOmONAL  time 

iocaling  and  aoquirfng 

using  aKamative 

or  your  (Mn  time  to* 

(mminulee) 

(rt  minutes) 

a.  Go  (0  Informeeon  Cenisr/ 

shared  cclectton 

. 

b.  tdentHy  document 

c  Locate  document 

d  Obtain  document 

e  Photocopy  documant 

— 

— 

Of  omeone  eiie'e  time  IT  >*ou  aakad 
or  would  aifc  aome  elM  (0. 

1  Go  to  Information 

Canter 

. 

g.  identify  document 

...  _ 

h.  Locale  document 

1  Obtain  document 

_ 

f.  Photocopy  document 

— 

— 

Other  ooeti 

k.  Purchef#  document 
(e  g ,  user  charge. 

(mdoHars) 

(indoSars) 

eubecrfptlon.  etc.) 

$ _ 

% _ 

1.  Photoc^  decuman! 

d _ 

$ 

m.  Teleph^  ca»i 

n.  Other  costs  (pkrase 

$ _ 

$ _ 

•PwaW 

% _ 

% _ 

These  questions  arc  primarily  analyzed  for  readings  of 
d-'cetnerts  rtbiaioed  from  the  information  center.  The  first 
question  above  iiidicates  how  much  information  would  be 
lost  if  there  was  no  informa'lori  center  and  v/hat  alternative 
sources  to  it  might  be  used.  The  second  question  (1st 
column)  indicates  the  amount  of  time  that  professionals 
(and  infonnation  center  users)  .spend  In  identif^ng,  locating 
and  obtaining  documents.  This,  added  to  time  spent  reading, 
is  an  indicfitor  of  the  “price”  paid  for  information  found  in 


documents  (and,  more  specifically,  documents  obtained 
from  the  information  center).  This  is  one  perspectiva  of  the 
value  of  information  provid^  by  information  centers  (i.e.,  a 
“willingness  to  pay”  value).  The  second  question  (2nd 
column)  determines  how  much  more  it  would  cost  uscis  to 
obtain  information  found  in  documents,  in  the  absence  of  an 
information  center.  TTiis  we  consider  to  be  the  principal 
estimate  of  the  value  of  information  centers.  Another 
perspective  of  value  is  found  from  assuming  that  users  spend 
a  relatively  fixed  amount  of  time  getting  and  using 
information.  The  second  question  shows  how  much 
additional  time  is  required  to  get  information,  if  there  was  no 
information  center  (see  Chapter  13). 

The  General  Population  Survey  provided  estimates  of  the 
hourly  rates  of  respondents  (or  they  may  be  obtained  in  the 
demographics  section  of  ih.e  Document 'Reeding  Survey). 
As  a  conservative  estimate  we  assume  that  “si  meone  else's” 
hourly  rate  is  one-half  that  of  users.  About  one-fourth  of  the 
lespondents  do  not  provide  responses  to  the  second 
question.  One  can  impute  directly  for  the  column  (i.e.,  actual 
lime).  This  means  that  non-respondents  are  assumed  to  take 
the  same  lime  as  respondents.  For  the  additional  time  spent 
using  alternative  sources  (column  2)  one  can  use  the 
response  for  alternative  source  as  the  basis  for  imputation. 
That  is,  calculate  average  time  and  costs  for  each  of  these 
alternative  sources.  If  the  respondent  answers  that  question 
but  not  additional  lime,  impute  the  appropriate  average. 
Otherwise  the  overall  average  is  used  to  impute.  Wc  find  that 
average  addiuonal  lime  is  typically  about  2  to  4  limes  higher 
than  acnial  lime  spent  locating  and  obtaining  the 
documents. 

It  is  useful  to  ask  readers  to  indicate  the  purpose  for  which 
the  document  was  read.  The  purpose  is  slated  the  same  way 
as  the  primary  work  activity  and  communications  aciivitie,s 
found  in  the  two  questions  soliciting  how  professionals 
spend  their  lime  at  work. 

Q  For  which  primary  ac'ivity  have  you  used,  or  do  you 
plan  to  use,  the  last  document  you  read?  Please 
indicate  the  one  principal  activity  that  is  most 
descriptive. 

Circle  ont  0060 


V  product  d»velopni9nt .  1 

Enginooring-reteafch  or  m«nuf«cturlr>0; 

procmirtg,  etc .  2 

Technical  or  research  support  (eg.,  quality  control,  standards. 

consumer  research,  etc) .  3 

Management  or  executive .  4 

Administration .  6 

Finance  or  accounting  .  6 

legaUpatent  .  7 

Medical .  3 

Operations  (eg.,  facilities  er^lneerlng,  mail  services,  etc.)  ...  9 

Background  research .  10 

Professional  development . . .  11 

Other  (piease  specify) _ _ _  •  •  •  12 


Q  For  which  communications  purposes  (if  any)  have  you 
used,  or  do  you  plan  to  use  the  last  document  you  read? 
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MMtf 


Circto  oo«  ood* 


Not  road  fo<  commurtication  purposas  . .  1 

Consisting  cr  ncMsIng  otiitra .  2 

MrV)r>g  iormai  ’eternal  pra^ontationa .  3 

Making  formal  a>1trnal  prasantaUons .  4 

standing  formil  internal  presentations .  5 

Attending  formal  external  presentations .  6 

Writing  (reports,  proposals,  cuticles,  etc.) .  7 

Reading  (reports,  proposals,  books. 

articles,  etc)  .  6 

Other  (please  soectM 


_ 9 

These  questions  provide  a  classification  for  the  contribution 
that  reading  and  information  centers  make  to  professionals 
work  (D  and  f  in  Figure  4). 


Then  one  can  “ask"  questions  that  relate  the  professionals' 
output  attributes  (F  in  Figure  4)  to  amount  of  reading  and 
source  of  documents  (f  in  Figure  4).  Prior  to  the  scries  of 
questions,  the  respondent  is  referr^  to  the  first  question 
above. 


Q  Did  reading  the  document  save  you  and/or  your  co¬ 
workers  any  labor  time  or  other  resources? 


ym  . 

No . 

Don't  know 


1 

2 

3 


We  find  that  about  20—30  percent  of  the  respondents  reply 
that  they  do  not  know  or  don’t  respond  at  all.  In  order  to  be 
conservative  in  our  estimates  we  normally  classify  these 
responses  as  “no”.  Generally,  we  find  that  about  25—35 
percent  of  journal  article  readings,  40—70  percent  of  book 
reaomgs  and  50—75  percent  of  report  readings  result  in 
such  savings.  Readings  of  information  center  documents  are 
always  higher,  particularly  for  journal  articles. 


0  Which  reason  best  describes  how  you  saved  (or  will 
save)  time  or  money  by  having  read  the  last  document? 

Ciicle  alt  that  apply 


Avoided  having  to  do  soma  woik  ........  1 

Pfovtded  conftrmallon  of  woik  in  progress  ...  2 

Stopped  an  unproductive  line  of  work  0 

Modified  an  activity,  project,  etc  . .  4 

tnitiated  a  new  activity,  project,  etc  . ,  .  5 

Other  (please  specily) _ 

_ ~  6 


The  questions  below  deal  only  with  the  principal  activity 
given  in  QX  above, 

0  Did  reading  the  document  result  in  performing  the 
activity  with  greater  quality? 


Ooni  know .  3 

Doesn't  apply .  . 

Typically  we  find  that  there  is  indicated  to  be  greater  quality 
for  about  40—60  percent  of  journal  article  readings,  70—80 
percent  of  book  readings  and  60—75  percent  of  report 
readings.  Documents  obtained  trom  information  centers 
tend  to  ha-  .•  much  higher  proportions  of  readings  of  higher 
qi'aiiiy. 

Sometimes  we  obtain  an  indicator  of  the  extent  of 
incr^se  of  quality  by  asking  respondents  to  rate 
quality  on  a  scale  of  one  to  seven  before  and  after  the 
document  is  read.  The  improvement  in  quality  is 
typically  1.5  to  2.0  times  the  initial  amount  (found  by 
dividing  the  average  rating  after  by  the  average  rating 
before). 

Q  Did  reading  the  oocumerit  result  in  your  performing  or 
completing  the  activity  faster  (in  less  duration  of  time)? 


Doni  know .  3 

Doss  not  apply .  .  . 

Here  we  find  much  lower  positive  responses.  Improved 
timeliness  tends  to  res"lt  from  about  20—30  percent  of 
journal  article  readings,  40—50  percent  of  book  readings, 
and  50—60  percent  of  report  readings.  Information  center 
documents  are  again  much  more  likely  to  be  favorable. 

The  literature  provides  a  nuinber  of  anecdotes  concerning 
how  reading  of  information  center  matenals  results  in  large 
savings  to  readers.  We  have  attempted  to  establish  the  extent 
to  which  such  savings  are  achieved  across  all  reading.  This  is 
done  through  a  series  of  five  questions.  These  questions 
relate  to  (E)  in  the  framework  Figure  4. 


Avoiding  having  to  do  some  work,  modifying  an  activity  or 
project,  and  providing  confirmation  of  work  in  progress  are 
the  most  frequently  cited  reasons  (about  30—50%  of 
readings  in  which  there  are  savings)  for  savings. 

Q  Considering  ONL'Y  direct  salaries,  what  is  the 
approximate  dollar  value  of  the  time  you  and/or  your 
co-workers  saved? 

S _ total  savings 

0  How  many  co-workers  were  involved  in  the  savings? 

_ co-workers 

Q  What,  if  any,  were  the  dollar  savings  achieved  for  other 
thing.s  (such  as  equipment,  supplies,  avoided  need  for 
consultant,  avoidance  of  regulatory  penalties,  etc.)? 

S _ total  other  savings 

Some  respondents  answer  the  earlier  questions  above,  but 
not  these  latter  questions.  We  normally  impute  average 
responses  to  these  item  nonresponses.  However,  one  could 
impute  based  on  the  reasons  for  savings.  The  question  of 
how  many  co-workers  aie  involved  is  asked  because, 
presumably  each  co-worker  has  a  chance  of  answering  the 
question.  Thus,  the  estimated  savings  should  be  divided  by 
the  number  of  co-workers  involved.  We  normally  estimate 
average  savings /or  a//  readings,  not  just  those  ui  which  some 
savings  are  incurred.  Savings  from  readings  from 
information  center  documents  are  typically  at  least  50 
percent  higher  than  for  documents  obtained  from  other 
sources  (especially  journal  articles). 

One  problem  with  interpreting  these  averages  is  that  one 
tends  to  think  of  average  as  being  “typical”.  The  distribution 
of  savings  from  journal  article  readings  from  one  company  is 
shown  ill  the  figure  below.  From  the  figure  it  is  clear  that 
about  two  percent  of  the  readings  contribute  to  nearly  all  the 
savings  achieved. 

Other  beneficial  consequences  are  also  sought. 
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Q  Please  indicate  any  other  benefits  that  you  have 
derived  from  reading  this  document. 


Orel*  that  ipply 


No  otlwr  banoflu  that  I  can  think  of .  1 

Rainforcad  hypothesaa  or  conlldanca  In  my  work .  2 

tnitlatad  klaaa  lor  my  work  .  3 

Halpad  gulda  futura  work .  4 

Brndanad  or  narrowad  options  concerning 

mywork  .  .  s 

Providad  naadad  markat  inialiiganca  about 

oompatltor(t) . ' .  8 

Used  In  lacturas,  samlnart,  ate  .  7 

Othar(spaclly) _ 

_  ...  8 


Usually  about  ten  to  25  percent  of  respondents  can  think  of 
other  benefits.  “Helped  guide  future  work”  is  the  most 
frequent  other  benefit  mentioned.  However,  all  other 
benefits  are  frequentiy  mentioned  as  well  (20—40%  of 
readings). 

Finaily,  one  can  determine  if  the  readers  knew  about  the 
information  reported  or  discussed  in  tne  document  prior  to 
their  reading  about  it. 

Q  Did  you  know  about  the  information  reported  or 
discussed  in  this  document  prior  to  reading  about  it? 


Yes  .  ' 

No  .  2 


The  information  found  in  journal  articles  is  much  less  likely 
to  be  familiar  to  readers  (25—50%),  than  for  books  and 
reports  (50—80%). 


Proportion  of  Journal  Readings 
In  Which  Various  Levels  of  Savings 
are  Achieved 


Proportion  of  Readings 

BOS  - - - ^ - 

rs.ss 


dOUACEi  KlAQ  Rtit«reh,  tne. 
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Chapter  11 

Relating  Information  Center  Performance  to  Effectiveness 


11.1  Background 

In  Chapter  3  we  gave  examples  of  several  ways  that  one  can 
express  the  relationship  between  input  cost  and  output 
quantities  and  attributes.  In  this  chapter  we  provide  some 
examples  for  relating  output  attributes  and  service 
effectiveness.  In  particular,  an  example  shows  that  amount 
of  use  is  related  to  level  of  satisfaction  with  a  service.  A 
further  example  shows  that  some  attnbutes  have  greater 
utility  than  others  and  that  one  can  set  a  relative  value  on 
attnbutes.  Since  satisfaction  with  attributes  is  demonstrated 
to  be  important,  we  give  an  example  which  relates  eenter 
input  cost  and  satisfaction.  Finally  we  give  other  examples  of 
how  one  might  relate  cost  to  other  effectiveness  measures. 

11.2  Relating  User  Satisfaction  and  Amount  of  Use  of 
Services 

An  example  is  given  helow  for  online  bibliographic  search 
services.  In  the  example  we  measure  quality  of  searches  by 
rating  user  satisfaction  with  ( 1 )  relevance  of  output  to  users’ 
information  needs  and  (2)  number  of  references  retrieved  in 
search  output.  Timeliness  is  observed  by  the  time  between 
request  and  receipt  of  search  results  and  it  is  measured  by 
satisfaction  ratings.  The  satisfaction  ratings  are  from  I  (very 
dissatisfied;  to  5  (very  satisfied)  An  example  of  satisfaction 
ratings  of  these  output  attributes  arc  given  in  Table  ll.l.Thc 
results  show  that  2.2  percent  of  users  are  very  dissatisfied 
with  relevance  of  output  to  users’  information  needs  and 
30.8  percent  of  users  are  very  .satisfied.  Average  ratings 
across  users  surveyed  is  4.15  or  just  above  4  (satisfied)  For 
this  orgamzation,  timeliness  is  rated  highest  in  average 
satisfaction.  The  question  becomes  what  effect  satisfaction 
has  on  amount  of  use  of  services.  Two  approaches  for 

TABLE  11.1 

EXAMPLE  OF  SA’nSFACTION  RATINGS  WITH 
ATTRIBUTES  OF  ONLINE  BIBLIOGRAPHIC  SEARCHES 


S«U$f«ction  Lfv«4* 


Aspects  o(Oniio« 
Bibliographic  S*«rching 

1 

(%) 

2 

(%) 

3 

{%) 

4 

5 

(%) 

Average 

Satisfaction 

Level* 

n«)«v*ncj  of  output 

To  Os«r«'  Informabon 

NMdt 

22 

03 

87 

580 

306 

416 

No  ofrafarancM 

in  March  output 

19 

29 

114 

54,7 

291 

406 

TVna  batwoan  raquaat 
and  rac^pt  of  March 
output 

17 

06 

96 

406 

515 

440 

5GunC£.  Kins  FwMaren,  me  Survey  of  Prof*Mion«i« 


*S«tit{*ctfon  SciJ*'  1  •  Vff  Di»Mtft6*d 

2  » 

3  •  N«ith«r  Satitfitd  nor 

4  >■  SattHM 

5  M  V*(y  Sutisflod 


examining  this  relationship  are  discu'  sed  below.  The  first 
approach  involves  correlating  number  of  searches  requested 
by  users  (per  year)  with  level  of  satisfaction.  Presumably 
users  vho  are  more  satisfied  will  use  the  service  more.  The 
seconu  approach  is  to  use  a  conjoint  measurement 
techniqt  a  developed  in  the  market  research  field.  In  that 
apprcdch  one  can  establish  the  relative  importance  of 
output  attributes. 

The  best  way  to  correlate  user  satisfaction  and  amount  of 
use  IS  to  obtain  both  measures  from  a  single  survey 
instrument  (see  Part  3)  One  can  calculate  the  aveiagc 
number  of  searches  requested  by  those  who  are  very 
satisfied,  satisfied,  etc.  Below  are  examples  for  three  search 
output  attributes.  For  relevance  of  search  output,  those  who 
are  very  satisfied  search  an  average  of  5.1 2  searches  per  year 
Those  who  are  satisfied  search  an  average  of  3.71  and  the 
average  number  of  searches  decreases  as  satisfaction 
decreases  to  0  43  searches  per  year.  Similar  results  arc  found 
for  number  of  references  in  search  output  and  timeliness  as 
well.  However,  the  seventy  of  decreases  in  number  of 
searches  .seems  to  be  less  than  for  relevance  for  the  other  two 
output  attributes.  Generally,  we  would  conclude  that  quality 
is  .somewhat  more  important  than  timeliness.  This  assertion 
IS  confirmed  in  the  example  below. 

Number  of  Searches  per  Year  by 
Level  of  Satisfaction  with  Relevance  of 
Output 

Number  of  Searches  per  Year 


6.12 


Vtry  aulllltd  NtllStr  DIaattlllica  Vtfy 

86IUII66  DI>Mlllll>6 

Level  of  Satisfaction 

SOURCEi  King  Raattreh.  Inc. 
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Number  of  Searche*  per  Year  by 
Level  of  Satlefactlon  with  Number  of 
Referencet  In  Search  Output 


Number  of  Searcheo  per  Year 


SOURCE  Kirifl  R«a««reh,  Inc. 


Number  of  Searches  per  Year  by 
Level  of  Satisfaction  with  Time  Between 
Request  and  Receipt  of  Search  Output 


Timeliness  can  also  be  specified  to  three  levels:  speed  of 
response  within  a  day,  between  one  and  three  days  and  over 
three  days.  Price  can  be  measured  at  three  levels:  S 100,  S 1 50 
and  $250. 

A  potential  model  for  assessing  pricing  of  bibliographic 
products  and  services  is  conjoint  measurement,  which  is 
used  in  marketing  research,  attempts  to  quantify  the  value 
systeir.j  of  users.  An  example  is  adapted  from  Johnson'% 
Supp'se  that  an  online  searcher  has  an  option  of  paying 
o'^rc  to  aehieve  greater  recall,  fewer  items  retrieved,  or 
faster  response  to  searches.  The  conjoint  measurement 
model  relies  on  users'  indicating  their  preferences  for 
different  prices  at  different  levels  of  relevance,  and  response 
times.  Conjoint  measurement  provides  a  relative  utility 
measure  of  search  attributes  such  as  quality  of  search,  speed 
of  re.-'ponse,  and  price.  Since  it  has  been  shown  that  the  value 
of  inforniaiion  center  services  is  in  saving  time  (a  scarce 
resource),  the  respondent  can  think  of  the  price  paid  in 
tciais  ot  their  time  as  well  as  in  user  charges. 

The  mathematical  method  requires  respondents  to  rank 
pairwise  combinations  of  attributes  using  the  questions 
below. 

O  In  the  matrix  below  are  two  attributes:  quality  of  search 
(i.c.,  high,  medium  and  low  relevance)  and  price  ($  100, 
SI50.  $250).  Please  rank  the  nine  combinations  of 
levels  from  I  to  9.  Presumably,  the  highest  rank  (1)  is 
high  relevance  at  a  cost  of  only  $100.  llic  lowest  rank 
(9)  is  low  relevance  at  a  cost  to  you  of  $250.  Please 
rank  the  remaining  cells  from  2  to  8. 


Number  of  Searehea  per  Year 


<.i« 


Very  titttfltE  Nflincr  OUtitUMcd  Vtry 
talUMH  OlmiUMtd 


Level  of  Satisfaction 

SOURCEi  King  Raaaarch.  Inc# 

The  conjoint  measurement  method  requires  users  of 
services  such  as  online  searching  to  make  judgments  about 
sets  of  alteinatives  involving  different  combinations  of 
search  output  attributes:  quality,  timeliness  and  price.  The 
output  attribute  of  quality  of  search  can  be  specified  at  three 
levels;  high,  medium  and  low  relevance  of  times  retrieved. 


Ouafity  ol  Saarch 
H<gh  relevance  of  items 
MeUium  ol  items 

Low  relevancf^  of  items 


Price 

$100 

$150 

$250 

1 

9 

Q  In  the  matrix  below,  we  give  two  other  pairs  of 
attnbutes:  speed  of  response  ^withln  a  day,  within  3 
days,  over  3  days)  and  price.  Please  rank  the  unranked 
cells  from  2  to  8. 


Speed  of  Response 
a  Day  .  .. 

Within  3  days . 

Ovef  3  days. .  .  . 


Pile* 

$100 

$150 

$250 

1 

9 

Q  In  the  matrix  below,  we  give  the  last  combination  of 
pairs  of  attnbutes;  quality  of  search  and  speed  of 
response.  Please  rank  the  unranked  cells  from  2  to  8. 


100 


QUAUIY  OF  SEARCH 


Sp*«d  Of  AMpooM 

Hgh 

Motfum 

LOW 

R«ievanc« 

R*l*v«ne« 

Within  •  0«y . . 

. 

Wnhlhdd<y» . 

Ovtr  3  d«yt . 

Consider  online  searches  differing  only  in  price  and  speed  of 
response  and  suppose  a  respondent  were  to  state  rank  order 
of  preference  for  online  searches  with  nine  combinations  of 
price  and  speed  of  response.  Such  data  could  be  arranges,  as 
follows: 

Pile* 


Sp««d  of  n«spon4« 

$100 

$tso 

$2S0 

Within  t  D<y . 

3 

7 

WUhtn  3  d«y9 

....  2 

4 

e 

Ov«r  3  dty»  . 

.  .  5 

6 

9 

If  these  data  were  examined  one  attribute  at  a  time,  it  would 
be  concluded  that  this  respondent  prefers  lower  prices  to 
higher  prices  and  faster  responses  to  slower  responses,  other 
things  being  equal.  Although  one  can  obtain  such  potentially 
valuable  information  by  examining  these  one  can  see  that 
although  this  respondent's  preferred  online  search  will 
require  a  $100  price  and  response  within  a  day,  his  second 
choice  shows  that  he  would  rather  drop  to  a  response  time  of 
within  3  days  rather  than  pay  the  higher  pnee  of  $  150.  Tims, 
by  considering  these  two  attributes  jointly,  one  can  learn 
something  about  their  relative  importance.  To  investigate 
this  respondent’s  value  system  more  generally,  we  could 
have  the  respondent  express  preferences  for  online 
searching  differing  in  relevance  and  timeliness  in  relevance 
and  price,  and  so  on,  If  the  user  were  very  highly  motivated, 
one  could  ask  the  user  to  provide  tradeoff  data  for  all 
possible  pairs  of  attributes  of  interest. 

A  numerical  example  is  given  to  show  how  conjoint 
measurement  can  be  used  to  infer  user  values  from  pairwise 
tradeoff  data.  Suppose  that  online  searching  could  be 
described  adequately  in  terms  of  four  attributes,  each  with 
three  levels  as  shown  above.  Rank  orders  of  preference  data 
for  an  actual  respondent  arc  shown  in  Table  11.2,  in  which 
are  shown  six  tradeoff  matrices,  one  for  each  pair  of 
attributes. 

Consider  a  simple  model  of  preference  formation  that 
assumes  that  each  respondent  has  a  positive  utility  value  for 
each  level  of  each  attribute,  and  that  the  relative  degree  of 
liking  for  a  specific  online  search  is  obtained  by  multiplying 
together  his  utilities  for  the  attribute  levels  describing  that 
online  search.  If  a  respondent’s  uhUties  are  known  for  the 
relevant  attributes,  one  could  predict  the  user's  rank  order 
of  preference  for  specific  online  search.  A  set  of  utilities  for 
such  a  set  of  responses  is  provided  in  Table  11.3. 

Relative  liking  for  an  online  search  of  medium  relevance 
with  a  price  of  $100  would  be  .33  X  .51  —  .1683.  This  is  a 
relative  value,  and  it  will  have  meaning  only  when  compared 
with  other  similarly  derived  values  for  online  searches 


having  other  levels  of  price  and  levels  of  relevance.  For  this 
person,  high  relevance  and  a  price  of  $150  would  have  a 
relative  value  of  .57  X  .34  —  .1938.  Therefore,  this  user 
would  prefer  the  latter  combination  of  price  and  relevance. 
In  choosing  among  online  search  capabilities  differing  in  all 
four  attributes,  the  user’s  relative  values  would  be  obtained 
by  computing  the  products  of  four  utility  values  at  a  time 
rather  than  two  at  a  time. 


TABLE  11.2 

ONE  RESPONDENT’S  TRADEOFF  DATA  (RANK 
ORDERS  OF  PREFERENCE) 


AvoriQe 
RMpOnSA  TiTW 

Averaga 

FMavance  Rama  BaMtvad 

<S  6>15> 

IS 

20- 

•OC.  B9C 

sec. 

rtflh 

Medium  Low  <20  50  >50 

Prico 

$100  1  2 

5 

1 

4  7 

1  3 

4 

$150  3  4 

6 

2 

5  8 

2  5 

6 

$200  7  6 

9 

3 

8  9 

7  8 

9 

Top  Sp»«d  of  n«spon»* 

Within  0 

1 

3  7 

1  2 

5 

Within  3  days 

2 

5  8 

3  4 

6 

Ovar  3  days 

4 

6  9 

7  0 

9 

Ralavanca  (%) 

Low 

1  4 

5 

M«dlum 

2  6 

7 

Htgh 

3  8 

9 

TABLE  11.3 

EXAMPLE  CF  ESTIMATED  UTILITY  VALUES  FOB  ONE 

RESPONDENT 

Level 

UtiSty 

Avsfaga  Reiovance 

High  Relevance 

57 

Medium  Relevance 

33 

Low  Relevance 

10 

PflCO 

$100 

51 

$150 

34 

$250 

IS 

Avaraga  Basponsa  Time 

Within  a  day 

31 

Within  3  days 

42 

Over  3  days 

27 

Utilities  are  estimated  so  as  to  account  simultane''usly  for  all 
six  of  his  pairwise  tradeoff  matrices  in  Table  1 1 .2.  By  way  of 
illustration.  Table  11.4  indicates  the  computations  of 
pairwise  products  for  the  price  versus  the  response  time. 
This  user’s  utihties  for  the  three  price  levels  are  shown  at  the 
top,  and  the  utilities  for  the  three  levels  of  r->levance  are 
shown  at  the  left  margin.  The  value  in  each  cell  is  obtained  by 
multiplying  together  the  utilities  for  that  row  and  ,;o(imin. 
The  rank  orders  of  the  numerica'  values  in  the  cells  of  this 
table  are  indicated  by  the  numbers  in  parentheses.  It  is  found 
that  these  pairwise  products  have  nearly  the  same  rank 
order  as  the  data  themselves,  the  single  exception  being  the 
cells  ranked  6  and  7.  Thus,  the  estimated  utilities  are  quite 
consistent  willi  the  data  and  may  be  taken  as  a  summary. 
These  utility  values  are  meaning^l  only  in  a  relative  sense.  If 
one  were  to  raise  them  to  any  positive  exponent  (such  as 
squaring  them  or  taking  their  square  roots),  their  meaning 
would  be  unchanged.  Further,  since  their  absolute 
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magnitudes  are  arbitrary,  they  are  scaled  so  that  the  sum  for 
each  attribute  is  unity. 


TABLE  11.4 

PAIRWISE  PRODUCTS  OF  UTILITIES 


3100 

Prloe 

$150 

$250 

5t 

34 

IS 

Htgh 

57 

m 

(2) 

(5) 

2907 

1036 

0655 

Medium 

33 

(3) 

0) 

m 

1633 

1122 

.1495 

Low 

10 

(6) 

(S) 

(9) 

0510 

.0340 

0150 

Although  the  model  underlying  this  computation  is  a 
multiplicative  one,  it  is  not  different  in  any  important  sen.se 
from  additive  models  in  more  eommon  use.  By  taking 
logarithms  of  these  values,  one  could  get  new  values  for 
which  sums  rather  than  products  would  have  the  desired 
rank  orders.  Even  considering  the  arbitrariness  of  scaling 
conventions,  these  particular  utility  values  arc  not  unique, 
other  values  obtained  by  slight  modifications  of  them  will 
still  provide  pairwise  products  having  almost  the  same  rank 
order  as  the  data.  However,  if  the  respondent  had  reacted  to 
several  pairs  involving  each  attribute  and  the  respondent 
were  to  solve  simultaneously  for  utilities  best  fitting  all  his 
preference  data,  there  would  probably  be  a  unique  solution 
apart  from  scaling. 

The  relative  value  of  relevance  and  timeliness  is  also 
expressed  in  the  figure  below.  This  figure  shows  the 

Relative  Tradeoff  in  Value  to  User  of 
Online  Search  Relevance  and 
Turnaround  Time 

Value  <$) 


monetary  value  of  all  nine  combinations  of  levels  of 
relevance  and  speed  of  delivery.  If  search  results  arc 
returned  within  one  day,  but  relevance  of  items  retrieved 
drops  from  high  to  medium,  the  value  decreases  from  S250 
to  S150,  Similarly,  if  relevance  remains  high  but  response 
time  drops  from  within  one  day  to  between  one  and  three 
days,  the  value  drops  from  $250  to  $195.  Thus,  over  all 
combinations  of  these  two  performance  attributes,  it  is 
estimated  that  the  value  of  quality  is  greater  to  the  user  than 
the  speed  of  delivery,  although  both  are  appreciable.  One 
can  see  that  low  relevance  of  items  retneved  and  response 
limes  of  over  three  days  reduce  value  to  only  an  estimated 
$  10.  Clearly,  both  quality  and  timeliness  of  response  should 
be  kept  at  a  high  level  when  at  all  possible. 

11.3  Relationship  ofSearch  Time  and  Satisfaction 

It  is  useful  to  know  how  much  it  costs  to  achieve  high  quality 
of  online  bibliographic  searches  One  can  evaluate  this 
relationship  of  online  bibliographic  searches.  One  ean 
evaluate  this  relationship  by  measuring  search  time  (say  in 
minutes)  and  quality  or  user  satisfaction.  For  an  example  we 
give  ratings  of  user  satisfaction  with  relevance  of  output  as  a 
measure  of  quality.  Ratings  of  satisfaction  arc  given  numbers 
from  1  —  very  dissatisfied  to  5  —  very  satisfied.  In  order  to 
establish  a  valid  relationship  ue  must  measure  time  and 
satisfaction  ratings  for  the  same  searches.  Time  can  be 
recorded  by  searches  at  the  time  a  search  is  performed  and 
users  can  be  asked  for  their  satisfaction  rating  on  a  search 
follow-up  form  or  by  contacting  the  user  by  mail  or 
telephone.  Hypothetical  data  arc  given  in  Table  h.5  for  a 
sample  of  50  online  searches. 

In  the  table  the  first  search  took  42  minutes  and  the  user 
rated  satisfaction  with  relevance  of  output  at  3  (neither 
satisfied  nor  dissatisfied),  the  second  search  took  108 
minutes  and  the  user  rated  satisfaction  as  5  (very  satisfied), 
and  so  on  The  average  time  required  per  search  is  71.9 
minutes  and  the  average  satisfaction  rating  is  3.56  (i.e., 
about  halfway  between  neither  satisfied  nor  dissatisfied  and 
satisfied,  a  generally  low  rating). 

Below,  search  time  is  plotted  against  satisfaction  ratings.  In 
this  example,  satisfaction  with  relevance  of  output  increases 
as  more  time  is  taken  to  search  as  one  would  expect.' 
However,  the  increase  in  satisfaction  becomes  smaller  with 
increases  in  time.  That  is,  increases  in  time  of  20  minutes 
from  20—39  to  40—59  minutes  shows  an  increase  of  1.10  in 
satisfaction  ratings  ( 1 90  to  3.00),  but  increases  in  time  of  20 
minutes  from  80—99  to  100—120  minutes  shows  an 
increase  of  only  0.40  in  satisfaction  ratings  (4.30  to  4.70). 
One  can  also  assign  dollar  values  to  the  searches  in  terms  of 
searchers'  time,  communication  costs,  hit  rates,  etc. 

Searcher  attributes  can  be  examined  as  well.  For  example, 
suppose  the  25  searches  in  the  first  two  columns  of  Table 
11.5  are  done  by  searchers  having  subject  knowledge  (e.g., 
chemistry,  engineering,  law,  etc.)  or  a  great  deal  of 
experience  and  the  other  25  searches  are  done  by  searchers 
with  no  subject  expertise  or  little  experience.  The  searchers 
with  subject  expertise  average  67.4  minutes  per  .search  and 
experience  average  satisfaction  ratings  of  4.04.  The 
searchers  with  no  subject  expertise  average  76.4  minutes  per 

'  In  fact,  real  factors  mitigate  this  relationship  somewhat.  For 
example,  vaguely  defined  searches  would  require  more  time  on 
the  part  of  searchers,  yet  may  even  so  result  in  relatively  low 
satisfaction  ratings. 
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search  and  average  satisfaction  ratings  of  3.08.  Thus  the  first 
group  of  searchers  is  mote  productive  and  search  uith 
greater  quality.  More  is  said  about  this  example  in  the  last 
chapter.  Also,  this  example  is  used  in  Chapter  7  to  describe 
methods  of  calculating  statistical  standard  errors  and 
confldence  intervals. 

TABLE  11.5 

EXAMPLES  OF  VALUES  FOR  DETERMINING 
RELATIONSHIPS  BETWEEN  TIME  SPENT  SEARCHING 
AND  RATINGS  OF  SATISFACTION  WITH  RELEVANCE  OF 
OUTPUT:  50  ONLINE  SEARCHES 


TIME 

(IMAGES) 

SARSFACnON 

RATMQ 

TIME 

(MMUTES) 

SATtSFACnON 

RATING 

42 

3 

78 

3 

100 

5 

66 

3 

32 

2 

46 

2 

2t 

2 

89 

4 

04 

4 

56 

2 

61 

4 

M8 

5 

MO 

8 

84 

4 

t2 

4 

34 

t 

103 

5 

76 

4 

72 

8 

66 

4 

31 

3 

59 

3 

86 

8 

36 

2 

63 

4 

114 

4 

112 

8 

74 

3 

61 

4 

97 

3 

86 

$ 

39 

I 

04 

8 

107 

4 

63 

$ 

79 

3 

33 

2 

56 

2 

63 

8 

67 

8 

62 

4 

120 

5 

118 

8 

37 

1 

46 

3 

M8 

4 

11.4  Examples  of  Relationships  Involving  Input  Cost, 
Effectiveness,  and  Domain  Measures 

Below  are  a  few  examples  where  cost  is  related  to  certain 
output  attributes  or  effectiveness  measures.  Some  of  the 
examples  are  trom  Keys  to  Success. 

Satisfaction  with  Relevance  of  Search 
Output  by  Search  Time 

Averago  Satisfaction  Rating* 

0  - 


47 


19 

3 

39 

— 

43 

1 

, 

20*39 

40*59 

60-79 

60-99 

100-120 

Search  Time  (Mint) 

SOURCE:  KInp  Ataeareh,  Inc. 

•Satlafaetion  Ratings: 

1  ‘Very  Oissatitfisd,  6  'Vary  Sstisfisd 


The  Amount  of  Use  By  Attribute  Levels  is  an  indicator  of 
how  the  various  attributes  of  output  (quality,  timeliness, 
availability,  accessibility,  etc.)  affect  the  amount  of  use  of  a 
service  or  product.  An  example  is  the  amount  of  use  of 
online  searching  and  output  attributes  (relevance, 
timeliness,  etc.),  discussed  earlier.  By  also  relating  cost  per 
use  to  levels  of  attributes  a  manager  can  predict  the  likely 
ch'ects  of  changes  in  how  much  it  will  cost  to  achieve  the 
desired  levels  of  attributes 


BOOSS  op  AVAIUBILXTY 


This  example  shows  that  amount  of  use  (and  average  use)  of 
a  service  goes  up  as  the  number  of  hours  of  availability  of  the 
service  increases.  Yet,  in  the  example,  increases  m  hours  of 
availability  up  to  a  point  do  not  increase  amount  of  use  very 
much. 


EXAMPLE  DISPIAY 


BOORS  or  AVAIUtBIUTY 

Thi  oove  example  displays  cost  per  user  across  number  of 
hours  of  availability  of  a  service  or  the  entire  information 
center.  In  a  previous  example  it  was  shown  that  total  amount 
of  use  (and  average  amount  of  use)  increases  as  hours  of  that 
total  amount  of  use  (and  average  amount  of  use)  increases  as 
hours  of  availability  increase.  Presumably  the  number  of 
users  also  goes  up  with  increased  availability  since,  for 
example,  some  users  niay  be  able  to  visit  the  center  and  use 
its  services  only  at  odd  times.  It  may  be  that  average  cost  per 
use  may  decrease  some  as  hours  of  availability  increases  up 
to  a  point  and  then  increase  rapidly  because  costs  may  be 
higher,  say  over  a  weekend  or  at  night,  and  additional 
number  of  users  may  not  be  very  high  during  weekends  or  at 
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night.  'I'hus,  cost  per  use  might  increase  with  increased  hours 
of  availability:  for  example,  about  70  hours  of  availability. 
Regardless,  this  indicator  suggests  the  value  of  increased 
hours  of  availability. 

The  Cost  Per  Capita  relates  the  amount  of  funds  (or  other 
resources  applied)  to  provide  a  service  to  the  population 
served.  The  assumption  underlying  this  indicator  is  that  the 
higher  the  Cost  Per  Capita,  the  better  the  information  center 
is  performing  (or  the  center’s  funders  are  performing). 
However,  this  comparison  always  assumes  that  the  Cost  Per 
Use  remains  constant  across  the  libraries  being  compared; 
otherwise,  the  indicator  could  be  negatively  correlated  with 
performance.  It  is  seen  as  an  indicator  of  the  investment 
being  made  on  behalf  of  the  service  population. 

EXAMnf  CXSPLAV 
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The  Cost  By  Satisfaction  Levels  indicator  relates  the  cost  of 
resources  to  provide  a  service  and  the  satisfaction  of  users 
with  that  service.  It  is  similar  to  Cost  By  Attribute  Levels, 
except  from  two  different  perspectives.  Cost  by  Satisfaction 
Levels  is  useful  in  helping  the  library  manager  to  understand 
the  resource  implications  of  improving  the  output  of  a 


service.  For  example,  the  indicator  can  be  used  to  predict 
what  resources  would  be  necessary  to  increase  satisfaction 
with  quality  of  a  service;  similarly  it  can  be  used  to  prediet 
the  effect  on  satisfaction  of  chanjang  the  resources  allocated 
to  the  service. 

The  relationship  of  this  indicator  to  performance  is  negative 
in  that  the  lower  the  cost  the  better  the  performance. 
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Average  input  costs  by  level  of  satisfaction  is  similar  to  Cost 
by  Attribute  Levels.  Thus,  another  kind  of  display  is  given  in 
this  example.  Here  average  cost  is  given  for  two  methods  of 
service  provision,  e.g.,  screened  online  search  output  and 
non-screened  output,  across  levels  of  satisfacnon.  Cost  of 
Method  B  is  less  than  Method  A  at  low  levels  of  satisfaction 
but  more  at  high  levels  of  satisfaction.  If  high  satisfaction  is 
important,  then  Method  A  is  superior,  even  though  overall 
average  costs  could  be  greater  for  Method  A 
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Chapter  12 

Cost  and  Benefit  Analysis  of  Information  Center  Service 


12.1  Background 

The  raison  d  etre  of  cost  and  benefit  analysis  is  to  assist  in 
making  decisions.  Such  decisions  might  be  about  an 
information  center  in  its  entirety,  a  function  performed  by 
the  center,  a  service  or  product,  an  activity  or  even  a 
source.  For  example,  decisions  might  concern  whether  to 
(1)  hire  a  person  (resource),  (2)  regularly  screen  output  of 
online  searches  (activity),  (3)  employ  an  online  cataloging 
system  or  a  manual  system  (function),  and  the  like.  The 
answers  to  such  decisions  almost  always  involve 
comparisons  of  alternatives,  and  the  costs  and  benefits  of 
these  alternatives  are  the  bases  upon  which  to  make  these 
comparisons.  What  we  mean  by  compansons  is  that  a 
service,  system,  etc.  bemg  evaluated  is  compared  with  some 
alternative.  For  example,  an  online  bibliographic  search 
service  would  be  compared  with  manual  searching, 
Marching  by  a  broker  or  other  such  alternative. 
Comparisons  would  be  made;  with  regard  to  input  costs 
output  quantities  (and  attributes),  effectiveness,  and  so  on.  ’ 

Costs  and  benefits  should  be  described  in  terms  of  the 
unfavorable  (i.e.,  costs)  and  favorable  (i.e.  benefits)  results  of 
the  comparisons.  For  example,  if  an  alternative  to  a  search 
service  is  more  expensive  than  the  search  service  the 
comparison  is  favorable  and,  therefore,  is  a  “benefit.”  If  on 
the  other  hand,  quality  of  the  service  is  lower  than  an 
alternative,  the  comparison  is  unfavorable  and,  therefore,  is 
a  “cost.”  The  term  “cost”  may  be  used  in  two  ways.  The  most 
common  use  is  in  the  application  of  resources  in  which  cost 
could  be  the  money  associated  with  the  resources  The  use 
of  the  term  in  this  chapter  would  preferably  be  “detriment” 
which  is  the  opposite  of  “benefit.”  However,  corvention  has 
been  to  use  the  term  “cost"  rather  than  “detnment.” 

One  can  describe  the  results  of  compari.son  at  all  levels  of 
input  and  output,  and  consequences  such  as  effectiveness 
and  higher  order  effects  mentioned  in  previous  chapters.  If 
the  comparison  is  unfavorable  at  any  of  these  levels  it  is 
recorded  in  a  cost  column.  If  the  comparison  is  favorable,  it 
is  recorded  in  a  benefit  column.  The  trade-off  between  cost 
and  benefit  can  be  made  by  comparing  all  the  items  in  the 
cost  column  against  the  items  in  the  benefit  column.  It  is 
emphasized  that  comparisons  of  a  service  with  alternatives 
can  be  described  in  any  of  the  evaluation  measures 
discussed  thus  far  (e.g.,  dollar  input  costs,  relevance  of 
search  output,  satisfaction  ratings,  etc.).  However, 
comparisons  can  also  be  in  non-quantifiable  terms  as  well' 
such  as,  the  equipment  reliability  is  higher  or  lower  or  ease 
of  use  is  better  or  worse.  Thus,  the  results  of  each  alternative 
can  be  describ^  in  terms  of  their  costs  (i.e.,  dollar  amounts 
associated  with  resource  expenditures  and  other 
quantifiable  and  non-quantifiable  detriments  that  occur  as 
well),  and  their  benefits  (i.e.,  dollar  amounts  associated  with 
the  favorable  outcomes  or  consequences  of  choosing  an 
^ternative  and  other  quantifiable  or  non-quantifiable 
benefits).  Costs  and  benefits  (of  a  resource,  activity,  service 
or  product,  function  or  the  entire  information  center)  are 


expressed  in  tenns  of  input  expenditures  and  other 
r^ources,  and  output  in  terms  of  quantities  and  attributes 
effectiveness  and  higher  order  effects.  However,  costs  and 
benefits  are  not  measured  as  direct  outcomes  or 
consequences,  but  rather  in  terms  of  comparisons  of 
resources,  activities,  and  so  on. 

In  summary,  costs  and  benefits  are  measured  by  determining 
the  unfavorable  or  favorable  companson  of  an  alternative 
component,  activity  or  service  or  product.  This  principle 
mid  some  other  concepts  of  cost  and  benefit  are 
demonstrated  through  two  examples;  the  first  involving  a 
resource  (cataloger)  and  the  second  a  service  (online 
bibliographic  searching).  These  examples  are  explored  after 
reviewing  some  concepts  of  cost  and  benefits  analysis. 

12.2  Review  of  Some  Cost  and  Benefit  Concepts 

To  begin  with,  one  must  specify  a  period  of  time  over  which  a 
cost  and  benefit  analysis  should  apply.  This  lime  period 
should  start  at  the  present  time  and  end  at  .some  specified 
time  in  the  future,  say,  three  or  five  years.  With  few 
exceptions,  past  costs  (or  benefits)  should  not  be  included  in 
a  comparison  or  analysis.  Expenditures  that  have  already 
been  incurred  are  the  results  of  past  decisions  and, 
therefore,  unless  such  decisions  are  being  evaluated 
retrospectively,  they  should  not  be  considered.  Relevant 
expenditures  he  in  the  future,  not  in  the  past.  Such  past 
expenditures  are  often  referred  to  as  sunk  costs.  The 
question  then  is  how  far  into  the  future  should  costs  and 
benefits  be  considered.  The  answer  to  this  question  depends 
on  how  far  into  the  ftiiurc  reasonable  estimates  can  be  made 
in  light  of  such  factors  as  inflation;  expected  amount  of 
input,  output  and  use;  and  the  like.  In  information  centers,  a 
typical  analysis  period  is  three  to  five  years  into  the  future. 

One  of  the  critical  aspects  of  deciding  whether  or  not  to 
incur  an  expenditure  is  to  compare  that  expenditure  against 
what  else  might  be  done  with  the  money.  In  that  sense  costs 
can  be  considered  benefits  foregone.  A  simple  comparison 
would  be  to  compare  the  benefits  gamed  from  investing  in  an 
automated  system  against  the  value  of  that  money  after  five 
years  if  it  were  pul  into  savings  or  an  investment.  The  value 
of  the  money  put  into  savings  will  increase  by  the  compound 
interest  (or  return  on  investment)  gained  in  the  future,  and 
will  decrease  by  the  loss  of  value  as  a  result  of  inflation 
Usually,  an  equation  that  incorporates  inflation  and  interest 
rates,  thereby  adjusting  future  costs  and  benefits  to  the 
current  dollar  value,  is  used.  Such  an  analysis  is  referred  to 
as  present  value  analysis.  It  is  particularly  important  because 
cost  (Md  benefit)  involves  different  kinds  of  expenditures 
including  direct  one-time  fixed,  direct  recurring  fixed  and 
direct  vanable  costs.  Indirect  costs  such  as  administration 
costs  might  also  be  allocated  to  resources  or  activities.  These 
concepts  were  discussed  in  Part  2. 

It  IS  important  to  remember  that  cost  (and  benefit) 
compansons  for  each  of  these  types  of  expenditures  are 
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likely  to  be  different.  For  example,  if  one  is  comparing  an 
existing  operational  system  against  a  potential  alternative, 
the  fixed  cost  of  the  existing  system  has  already  been 
incurred  (i.e.,  it  is  a  sunk  cost)  and  should  not  being  included 
m  the  comparison.  If  a  new  system  is  being  considered  {say, 
over  a  tive-year  planning  period),  the  present  value  of  fixed 
costs  would  be  calculated  over  the  entire  five  years.  The  first 
year  recurring  cost  would  be  considered  over  five  years,  the 
second  year  recurring  cost  would  be  considered  (at  an 
inflated  rate)  over  four  years,  and  so  on.  Variable  costs 
would  change  over  years  based  on  projected  activity  as  well 
as  projected  inflation  rates  and  interest  rates. 

Also,  we  find  that  the  costs  (and  benefits)  associated  with  the 
different  categones  of  functions  (i.e.,  library  operational 
functions,  user  related  functions  and  support  functions)  arc 
somewhat  different  in  that  variable  costs  of  operational 
functions  involve  quantities  input  or  produced  (eg., 
interlibrary  loans  processed  or  searches  performed) 
Support  functions  do  not  involve  vanable  costs  except  in  a 
very  indirect  way.  By  doubling  the  user  related  services,  we 
may  find  that  the  cost  of  support  functions  may  be  less  than, 
the  same  as,  or  more  than  double  the  costs.  This  results  in 
the  use  of  scaling  factors  which  have  been  the  subject  of 
considerable  analysis  in  recent  years. 

Indirect  costs,  such  as  those  associated  with  support 
functions,  may  be  allocated  to  the  object  of  evaluation  such 
as  a  service  and  an  alternative  to  which  it  is  compared.  Such 
allocation  usually  is  in  pioportion  to  the  totai  direct  costs 
involved. 

Another  aspect  of  costing  involves  those  costs  that  one  can 
attribute  directiy  to  a  resource.  A  person  or  item  of 
equipment  can  have  several  types  of  costs  directly  related  to 
them.  For  example,  an  individual  has  a  salary,  fringe 
benefits,  some  furniture,  work  space  and  so  on  that  can  be 
included  in  the  person's  rates.  With  equipment  such  costs 
could  be  insurance,  maintenance,  space  and  so  on.  Also, 
there  are  indirect  overhead  items  such  as  supervision  and 
administration  that  probably  ought  to  be  allocated  to  the 
direct  costs  (sec  Chapter  4).  These  are  indirect  costs  that  arc 
more  difficult  to  determine  how  to  handle  in  cost  and  benefit 
analysis.  As  a  general  rule,  it  is  best  to  include  as  many  direct 
costs  as  possible  (i.e.,  fringe  benefits,  rent,  etc.)  and  include 
indirect  costs  (i.e.,  overhead)  only  when  such  costs  arc 
unique  to  the  operational  level  being  considered,  but  not  to 
an  alternative  to  which  it  is  going  to  be  compared.  If  it  is 
common  to  both,  then  it  should  not  enter  the  cost  and 
benefit  analysis. 

One  final  concept  that  one  needs  to  understand  is  the 
difference  between  marginal  costs,  average  costs  and  total 
costs.  Total  costs  arc  the  sum  of  one-time  fixed,  lecurring 
and  variable  costs  (of  quantities  input,  output  or  uses). 
Average  costs  arc  the  total  costs  divided  by  quantities  input 
or  output,  or  number  of  uses.  Marginal  costs  are  the  costs  of 
marginal  cha.iges  whether  they  be  of  an  additional  item 
produced  or  in  terms  of  changes  in  components,  activities, 
services  or  products,  or  functions.  Cost  benefit  analysis 
involves  an  assessment  of  the  marginal  changes.  Benefits  are 
maximized  or  costs  are  minimized  only  when  the  marginal 
ratios  of  benefits  to  costs  are  equal  for  all  alternatives. 

12.3  An  Example  of  the  Costs  and  Benefits  of  Catalogers 

An  example  is  given  concerning  how  one  can  compute  the 
costs  and  benefits  of  a  resource  (or  activity,  sci  vice,  etc.)  by 


companng  that  resource  to  a  realistic  alternative.  This  will 
be  done  by  an  example  of  the  hiring  of  a  cataloger  to  do 
on^nal  cataloging.  Assume  that  the  decision  at  hand  is  to 
hire  a  specific  professional  with  an  MLS  who  will  work  full 
time  at  cataloging  at  a  base  salary  of  $20,000.  Assume  also 
that  the  person  will  stay  in  this  position  for  three  years  with  a 
salary  and  fnnge  benefit  increase  of  six  percent  per  year  over 
that  period  of  time  Further,  assume  that  the  alternative  is  to 
hire  an  inexperienced  person  (non-MLS)  at  a  base  salary  of 
$15,000,  but  who  can  be  trained  to  catalog  at  a  one-time 
cost  of  $5,000.  Table  12.1  summarizes  the  costs  and  benefits 
associated  with  this  decision. 

In  the  first  colnf”"  — " the  different  levels  at  which  costs 
and  bcnclits  of  hiring  thi  MLS  over  the  alternative  are 
measured  In  the  secopj  and  third  columns  we  list  the 
measures  or  statements  associated  with  each  of  the 
alternatives  Finally  in  the  last  two  columns  we  list  the  actual 
costs  and  perceived  benefits  of  hinng  the  MLS  over  the 
alternative.  Let  us  consider  each  level  of  measurement  in 
turn  below 

The  fixed  one-time  costs  in  this  example  are  the  training 
expense'..  We  assumed  that  the  MLS  would  require  training 
that  would  cost  about  $1,000  whereas  the  non-MLS  would 
require  considerable  more  training  that  would  cost,  say. 
$5,000.  At  this  level  the  benefit  of  hiring  the  MLS  over  the 
non-MLS  would  be  expressed  as  $4,000  since  the  expenses 
of  the  MLS  are  less  than  the  expenses  of  the  alternative  by 
that  amount. 

The  recurring  cost  estimates  are  derived  from  the  base 
salary  levels  of  the  MLS  and  non-MLS.  Their  respective 
salaries  ($20,000  and  $15,000)  arc  multiplied  by  a  factor  of 
50  percent  to  take  fringe  benefits  and  overhead  into 
account,  so  that  these  expenses  become  $30,000  and 
$22,500  per  year,  respectively.  Because  we  initially  set  our 
planning  period  at  three  years,  we  need  to  determine  how 
these  expenses  will  change  over  those  three  years.  Two 
factors  arc  taken  into  account;  inflation  or  salary  increases 
and  discounting  for  present  value.  This  is  done  by 
multiplying  by  an  inflation  factor,  say  six  percent  and 
dividing  by  a  discount  factor,  say,  ten  percent  Numcne 
calculations  arc  given  as  follows' 


Year 

1  2  3 


MLS 

mooo 

30.000  X  1.00 

1.1 

11x11 

Altamatlve 

$22,500 

22.500  X  1.06 

2?  500  XI  CSV  1.06 

1.1 

1.1  X  1.1 

The  three  year  sum  of  the  expenses  for  the  cataloger  with  an 
MLS  is  $86,767  (i.e.,  $30,000  -f  $28,909  +  $27,858)  and 
the  three  year  sum  of  expenses  for  the  alternative  is  $65,075. 
The  cataloger  with  an  MLS  would  cost  about  $21,692  more 
over  the  three  year  planning  period  so  that  this  difference 
would  end  up  in  the  cost  column. 

Cataloger  Characieriilics 

At  this  level  we  made  the  assumption  that  hiring  a  person 
with  an  MLS  over  an  inexperienced  non-MLS  would  give 
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lASLE  12.1 

SUMMARY  OF  COSTS  AND  BENEFITS  ASSOCIATED 
WITH  THE  DECISION  OF  WHETHER  TO  HIRE 
A  CATALOGER  WITH  AN  MLS  DEGREE 


Levels  of 

Measures 

Person  Being 
Hired 

Alternative 

Costs 

Benefits 

ExDenditures 

Fixed  one-time 

Fixed  Recurring 

$  1,CXM/Training 
$30.000A'ear 

$  5,000/Training 
$22.500A'ear 

$2f,692 

$4,000 

Characteristics 

MLS 

BR 

Better  com¬ 
petencies 
(knowledge, 
skills,  attitudes) 

Effectiveness 

Quality 

QuanWes 

Produced 

95%  Accuracy 

2,250  items/yr. 

85%  Accuracy 

1,500  items/yr. 

10%  more  accuracy 

2,250  more  items  prO’ 
duced  (i.e.,  750  per 
+  $32,538  year) 

Hloher  Order  Effects 

Better  user 
satisfaction:  more 
use;  increase  value 
to  organization 

SUM 

$21,692 

$36,538  -1-  10% 

more  accuracy 
+  better  user 
satisfaction  + 
more  use  + 
increased  value 
to  organization 


the  center  the  benefit  of  greater  cataloging  competencies 
(knowledge,  skills,  attitudes). 


The  assumption  we  made  for  this  example  concerning 
quantities  produced  were  that  the  MLS  could  do  origin^ 
cataloging  at  a  rate  of  about  1.25  items  per  hour  (i.e.,  2,250 
items  per  year,  assuming  an  1,800  hour  year,  and  the  non- 
MLS  only  1,500  per  year).  Once  again,  we  need  io  consider 
these  levels  of  activity  and  their  associated  expenses  over 
each  of  the  three  years  of  the  defined  planning  period.  The 
alternative  cataloger  approach  would  require  another 
person  one-half  time  to  do  the  original  cataloging 
perform'ed  by  the  cataloger  with  an  MLS,  The  cost  over  the 
three  year  planning  period  of  doing  this  would  be  $32,538 
($ll,250-FS10,841-l-$10,447).Thus,oneofthe  benefits  of 
having  an  MLS  cataloger  do  the  original  cataloging  would  be 
$32,538  savings  on  expenses. 


Higher  Order  Effects 

At  higher  levels  we  assume  that  better  cataloging 
competency  and  better  cataloging  performan.e  result  in 
higher  user  satisfaction  which,  in  turn,  results  in  more  use  of 
the  library.  In  turn,  we  have  shown  in  Chapter  1 3  tiiat  greater 
use  of  information  yields  greater  value. 

In  summary,  the  net  benefit  of  hinng  the  MLS  over  the  non- 
MLS  is  $14,846  plus  10  percent  more  accuracy,  more  items 
cataloged  per  cataloger,  greater  user  satisfaction  and  more 
use  of  the  center.  All  decision  alternatives  can  be  evaluated 
using  this  cost  and  benefit  balance-sheet  approach. 

12.4  .An  Example  cf  the  Costs  and  Benefits  of  Online 
Bibliographic  Searching  Services 

In  this  example  (from  a  company)  we  look  at  the  cost  and 
benefit  of  an  online  bibliographic  search'  service  that  does 
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about  4,500  in-depth  searches  per  year.  Users  are  asked 
what  alternatives  they  would  use  if  the  information  center 
could  not  provide  the  search.  The  search  would  not  be  done 
for  about  19  percent  of  the  searches.  For  the  searches  that 
would  be  done,  the  usets  said  that  38  percent  of  the  searches 
would  be  done  by  them  manually,  19  percent  done  by 
themselves  online  or  by  someone  else  on  the  .staff  (8%),  14 
percent  would  have  asked  another  information  center  to  do 
it,  15  percent  would  have  called  a  colleague  who  is 
knowledgeable  in  the  field,  or  6  percent  would  have  a 
contractor/outside  service  to  do  it.  TTie  costs  and  benefits  of 
the  online  service  are  determined  by  comparing  the  service 
with  these  alternatives  regarding  service  expenditures  (i.c., 
service  input  cost),  service  output,  effectiveness  and  higher 
order  effects.  Results  of  these  comparisons  and  costs  and 
benefits  are  given  on  Table  12.2. 

Tlie  expenditures  (input  costs)  include  fixed  recurring  costs 
for  terminals,  search  aids,  searcher  training  and  continuing 


education,  telephones,  and  rent.  These  expenditures 
amount  to  about  $17,000.  Reference  materials  come  to 
another  $9,000.  Variable  costs  include  communication 
charges,  database  use,  off-line  printing  and  list  rates  and 
searchers’  time.  This  amounts  to  about  $47.50  per  search  or 
about  $2 14,000  total.  All  of  these  expenditures  are  recorded 
in  the  cost  column.  Output  as  indicated  is  4,500  searches. 
However,  users  said  that  about  855  (19%)  of  the  searches 
would  not  be  done  as  an  alternative  to  the  service.  The  855  is 
recorded  in  the  benefits  column.  The  users  also  said  in  a 
survey  that  92  percent  of  the  searches  done  by  the  center's 
service  would  be  done  better  by  the  seivice  (than  an 
alternative),  85  percent  would  be  done  faster  and  87  percent 
resulted  in  saving  time  of  the  users.  These  results  are  placed 
in  the  benefits  column. 

Users  were  also  asked  to  indicate;  "How  much  time  was 
spent  by  you  or  someone  form  your  staff  in  the  in-depth 
search?”  and  “Approximately  how  much  more  time  would  it 


TABLE  12.2 

SUMMARY  OF  COSTS  AND  BENEFITS  OF  ONLINE 
BIBLIOGRAPHIC  SEARCH  SERVICES 


Levels  of  Measure  Online  Search  Least  Expensive  The  Cost  or  Benefit 

Alternative  of  Online  Service 


Costs 

Benefits 

Expenditures 

Fixed 

Reference 

$  17,000 

0 

$  17,000 

materials 

$  9,000 

0 

$  9,000 

Variable 

$214,000 

0 

$214,000 

Output 

Quantities 

Quality 

Timeliness 

Other 

4,500  searches 

3,645 

855  more  searches 
92%  better 

85%  faster 

87%  saved  time 

Effectiveness 

User's  time 

$  170,000 

$  760,000 

$  590,000 

Other  labor 

$  12,000 

$  92,000 

$  80,000 

Other 

$  52,000 

$  482,000 

$  430,000 

Hioher  Order  Effects 

Items  read 

80,000 

64,900 

215,1(X)  readings 

Consequences 

$5,530,000 

$4,480,000 

$1,050,000 

SUM  $240,000  $2,150,000 

92%  of  researches 
better,  85%  of 
searches  faster, 
87%  saved  time, 
15,100  more 
readings 
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have  cost  you  to  get  the  search  from  alternative  sources?" 
(that  they  specified  earlier  in  the  questionnaire).  It  turns  out 
that  us<  rs  currently  spend  about  $234,000  on  searches 
(users’  time,  other  labor  and  other  costs)  and  $1,330,000 
using  alternative  sources  to  the  center.  Thus,  we  record 
savings  of  $590,000  in  users’  time  ($131  per  search), 
$80,000  in  other  labor  ($18  per  search)  and  $430,000  in 
other  expel, ditures  such  as  search  fees,  telephone  calls,  etc. 
Thus,  their  total  benefits  came  to  about  $  1.)  million 
(compared  with  center  costs  of  $240,000)  or  a  4.6  to  1  ratio. 

Higher  order  effects  result  from  readings  from  the  search 
outputs  that  are  lost  because  users  would  not  do  855  of  the 
searches.  We  do  not  know  what  the  consequences  are  of 
poorer  and  less  timely  alternative  searches.  Users  indicated 


that  they  read  (or  intend  to  read)  about  17.8  items  per  search. 
Thus,  the  users  read  about  80,000  items  as  a  result  of  the 
4,500  searches.  Tliey  would  lose  about  15,100  items  read 
(i.e.,  19%  of  the  searches  that  would  not  be  done).  We  have 
found  that  the  value  of  those  readings  is  about  $69  per 
reading  (see  Chapter  13.4).  Therefore,  approximately 
$1,050,000  might  be  lo.st  in  value  from  these  readings  fhi  > 
amount  is  recorded  m  the  benefits  column. 

The  sum  of  costs  and  benefits  are  given  in  the  table.  Costs 
come  to  about  $240,000  and  benefits  are  $1.1  million  for 
direct  effectiveness,  $1.0  million  or  potential  higher  order 
effects,  855  more  searches  done,  92  percent  of  them  done 
’oetter  than  alternatives,  85  percent  done  faster  and  87 
percent  of  the  searches  saved  time  of  the  users. 
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Chapter  13 

The  Value  of  Information  Centers 


13.1  Background 

In  this  chapter,  we  explore  ways  to  assess  the  value  of 
information  centers.  Below,  we  briefly  discuss  the  rationale 
that  we  have  used  in  assessing  value  of  services.  In  particular, 
value  is  assessed  from  three  perspectives;  what  users  are 
willing  to  pay,  how  much  more  it  would  cost  users  to  get 
information  if  the  services  were  not  available,  and  the  extent 
to  which  the  services  achieve  cost  savings  for  the  users. Then 
we  analyze  the  value  of  the  information  centers  in  providing 
journals,  books  and  internal  reports.  We  also  assess  the 
value  of  online  bibliographic  searching  and  Current 
Awareness  Bulletins,  since  they  are  particularly  important 
services.  The  value  of  information  centers  is  found  to  be 
substantial,  regardless  of  the  perspective  from  which  the 
evaluation  is  performed. 

13.2  Value  of  Information  Center  Collection 

There  are  three  levels  that  are  considered  in  assessing  the 
contribution  that  an  information  center  makes  to  the  value 
of  information.  Information  centers  are  not  inexpensive. 
Typically,  organizations  spend  an  average  of  between  $500 
and  $1,500  per  professional  in  their  organization.  On  behalf 
of  the  organization,  the  center  pays  for  expensive  journals, 
books  and  other  materials.  Acquiring,  processing, 
maintaining  and  distributing  these  materials  in  a  timely  way 
IS  very  labor-intensive.  Furthermore,  information  centers 
provide  a  range  of  other  services  such  as  reference,  online 
searching,  translation,  and  so  on.  There  needs  to  be  a  clear 
demonstration  of  a  favorable  return  on  this  investment. 

The  fii't  question  that  comes  to  mind  is  whether  the  pnee 
paid  for  center  materials  and  services  has  a  concomitant 
value.  The  lowest  bound  for  asses.sing  this  value  is  from  the 
perspective  of  the  readers.  What  are  they  willing  »o  pay  for 
this  information?  One  can  readily  measure  what  they  do  pay, 
recognizing  that  they  might  pay  more  if  they  had  to.  Time  of 
professiora.s  is  a  scarce  resource.  Professionals  must  decide 
how  to  utilize  their  time  in  order  to  be  most  productive. 
Engineers,  sci  'iiu.’ts,  lawyers,  administrators,  and  so  on, 
devote  a  substantial  .imount  of  their  time  to  getting,  reading 
and  using  information  found  in  documents  such  as  journals, 
books,  internal  doeuments,  patents,  and  so  on.  Their 
decision  to  use  their  scarce  time  for  information  sees  mg  and 
reading  is  a  strong  indication  of  the  value  they  place  on 
mformation.  The  total  time  (and  the  dollar  amount 
represented  by  this  time)  spent  on  information  provided  by 
an  information  center  is  an  indicator  of  the  value  of  the 
center.  In  organizations,  this  value  tends  to  be  on  the  order 
of  several  times  that  of  the  cost  to  the  information  centers  in 
purchasing  and  providing  these  materials. 

Of  course,  the  information  could  be  obtained  by  the 
professionals  from  other  sources.  They  could  subscribe  to 
journals  or  puichase  books  themselves.  Then  they  would 
lo.se  potent!^  savings  achieved  by  sharing  these  materials. 
They  could  use  another  source,  such  as  an  academic  library, 
but  tliat  involves  substantial  professional  time  required  to 


identify,  locate  and  get  access  to  these  other  sources.  Tliey 
could  order  materials  from  document  delivery  services, 
publishers  or  elsewhere,  but  that  assumes  an  ability  of 
professionals  to  identify  needed  information  then  locate 
where  it  is  and  acquire  it.  Furthermore,  if  all  professionals 
relied  on  academic,  public  or  other  libraries,  these  libraries 
would  soon  stop  making  their  collections  available  because 
of  the  enormous  cost  and  possible  denial  of  access  to  their 
own  primary  patrons.  In  fact,  this  trend  has  already  begun  at 
some  academic  institutions  in  the  US.  TTie  point  is  that 
having  a  nearby  library  in  the  parent  organizations  saves 
their  professionals  considerable  time  and  money. 

About  two-thirds  of  the  cost  savings  achieved  by 
information  centers  involves  professional  lime.  We  have 
observed,  over  the  years,  that  professionals  tend  to  spend  a 
relatively  constant  proportion  of  their  time  in  getting  and 
reading  information.  The  amount  of  time  they  spend  may 
shift  from  accessing  to  reading  or  vice  versa,  but  the  total 
seems  not  to  vary  much.  With  this  in  mind,  we  developed  a 
rationale  for  determining  what  would  happen  if 
professionals  had  to  rely  entirely  on  non-information  center 
sources  (i.e.,  if  there  were  no  center).  We  assume  that  there 
would  be  less  reading  because  more  time  would  be  required 
for  identifying,  locating  and  acquiring  information  from 
other  sources.  Therefore,  the  potential  benefits  derived  from 
readings  that  are  lost  would  not  bo  achieved.  $uch  benefits, 
include  savings  (in  lime,  equipment,  etc.)  derived  from 
information  and  improvements  in  quality  of  work,  timeliness 
of  work  output,  and  so  on.  Such  lost  benefits  are  what  we 
consider  the  highest  order  of  value  of  the  information  center 
services.  This  value,  compared  with  the  cost  of  centers,  is 
substantial.  The  savings  alone  are  typically  found  to  be  on 
the  order  of  10  to  20  times  the  total  cost  of  the  cental 
services. 

Determining  the  extent  to  which  services  contnbulc  to  the 
value  of  information  is  achieved  using  the  following 
rationale: 

•  The  number  of  readings  that  are  made  from  materials 
provided  by  the  information  center  is  first  determined. 

•  The  amount  of  time  that  the  professionals  spend  in 
identifying,  locating,  obtaining  accessing  and  reading 
the  materials  provided  by  the  center  is  estimated.  This 
is  what  the  professionals  are  “willing  to  pay”  for  these 
materials. 

•  Then  assume  that  center  services  are  not  available  to 
the  professionals.  If  they  are  not  available,  the 
professionals  would  have  to  get  their  journals,  books, 
etc.  from  alternative  sources  such  as  personally 
subscribing,  using  other  office  collections,  going  to  an 
external  library,  contacting  a  colleague  to  get  materials, 
and  so  on.  Ever,  'ssuming  the  least  expensive  and  time- 
consuming  alternative,  we  find  that  professionals  must 
spend  more  of  their  time  getting  access  to  the 
information  and  (hat  additional  costs  are  involved  as 
well. 
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•  llie  additional  amount  of  time  and  other  costs 
required  by  professionals  is  estimated.  This  is  the 
second  level  of  the  value  of  the  information  center. 

•  It  is  assumed  that  professionals  spend  a  given  amount 
of  time  in  information  seeking  and  reading.  Because 
they  would  have  to  spend  more  time  if  center  services 
were  not  available,  they  would  read  less  and,  tlierefore, 
lose  savings,  timeliness,  quality,  productivity,  etc., 
resulting  from  lost  readings. 

•  The  dollar  savings,  quality,  timeliness,  productivity, 
etc,,  that  are  lost  by  not  having  the  center  available  are 
considered  to  be  the  third  type  of  value  of  these 
services. 

13.3  The  Value  of  Journal  A,  deles  Provided  by 
Infomutlon  Centers 

In  this  example,  professionals  from  the  organization 
surveyed  are  estimated  to  read  about  600,000  journal 
articles  per  year.  About  196,000  of  these  readings  are  from 
journals  provided  by  the  information  center  (from  copies 
located  at  the  center,  journal  routing,  etc). 

The  approximate  professional  time  required  for  identifying, 
gaining  access  to  and  reading  journal  articles  obtained  from 
the  center  is  estimated  as  given  in  Table  13.1  for  all  article 
readings  from  the  center  journal  copies. 

TABLE  13.1 

AMOUNT  OF  TIME  SPENT  PER  YEAR  IN  IDENTIFYING, 
LOCATING,  OBTAINING  AND  PHOTOCOPYING  JOURNAL 
ARTICLES  OBTAINED  FROM  INFORMATION  CENTER  BY 
ORGANIZATION  PROFESSIONALS 


ActMty 

Avg.  Time 

Per 

Reading 

(rrlnutee) 

Tot^ 

Amount 

of  Time  per  Year 
(OOO  hours) 

PrVl?5Slonfi|'s  Tims 

Going  to  center 

3.7 

12.2 

Identify  article 

1.4 

4.6 

Locate  aiticie 

3,4 

11.2 

Obtain  aitids 

02 

2.6 

Photograph  article 

U 

LS 

Total 

11.7 

38.S 

Someone  ^Ije’a  Tirpe 

Going  to  center 

Z\ 

6.9 

Identify  article 

1.4 

4.6 

Locate  article 

1.4 

4.6 

Obtain  article 

1.3 

4.3 

Pho'ograph  article 

Si 

Z6 

Total 

7.0 

23.0 

SOURCE;  King  FtoMwch,  Inc.  Suivey  of  ProfeMlonals 


The  estimated  total  amount  of  time  professionals  spend 
identifying  and  gaining  access  to  journal  articles  provided  by 
the  information  center  is  about  38,500  hours  per  year  or 
about  11.7  minutes  (0.195  hours)  per  article  read.  Adding  to 


that  the  amount  of  time  spent  rea  ig  (0.7  hours  for  journal 
readings  from  center  copies),  it  estimated  that  the  total 
time  spent  is  about  175,700  hours  per  year  or  $6.7  million 
for  the  professionals  (assuming  an  average  hourly  wage  of 
$38.13).  Adding  to  that  the  costs  of  the  time  of  others  and 
other  costs  ($2.20  and  $0.80  per  article  read  or  a  total  of 
$588,000)  yields  a  total  of  $7.3  million. 

The  average  current  cost  of  these  readings  of  journals 
provided  by  the  center  is  $37.20.  This  is  the  minimum  that 
profession's  are  “willing  to  pay"  for  these  matenals  and,  as 
such,  the  amount  is  a  lower  bound  on  the  value  of  journal 
articles  provided  by  the  center.  Typical  average  additional 
costs  to  the  center  of  purchasing  and  proc-ssing 
subscriptions  and  conducting  online  searches  are  estimated 
to  be  about  $12.00  per  reading.  Thus,  the  ratio  of  center 
costs  ($12.00)  to  this  value  of  information  ($37.20)  is  about 
three  to  one. 

There  are  two  ways  that  we  have  studied  the  methods  and 
costs  of  obtaining  journal  articles  from  alternative  sources 
(i.e.,  other  than  the  center).  The  first  involves  observing  from 
national  surveys  which  explore  how  scientists  and  engineers 
get  their  articles  when  they  have  no  library  available  (e.g., 
when  they  are  employed  by  small  businesses).  We 
determined  the  approximate  amount  of  time  spent 
identifying,  locating  and  getting  information  from  tl:e  other 
sources.  For  the  second  method,  we  also  asked  professionals 
to  indicate  (1)  how  they  would  obtain  the  information  (not 
necessarily  the  journal  article,  if  another  source  such  as  a 
colleague  or  consultant  was  less  expensive)  from  the  least 
expensive  alternative,  and  (2)  how  much  additional  costs 
(above  the  current  cost)  would  be  required  in  terms  of:  (1) 
their  time;  (2)  the  time  of  others  (e  g.,  secretary,  technician, 
etc.);  and  (3)  other  costs,  such  as  subscription  to  a  journal, 
etc. 

The  pioccss  began  by  determining  whether  the 
professionals  knew  about  the  information  reported  (or 
discussed)  in  the  most  recent  article  read,  prior  to  their 
reading  about  it.  About  68  percent  of  the  readings  involved 
new  information.  The  readers  were  asked  how  they  would 
get  the  articles,  if  the  information  center  could  not  be  used. 
About  four  percent  of  the  professionals  said  they  would  not 
obtain  the  article  or  information.  Of  the  readings  in  which 
the  information  was  not  new,  the  information  would  mu»t 
frequently  be  obtained  from  a  colleague  or  other  source. 

The  average  additional  costs  (i.e.,  how  much  more  it  would 
cost  over  current  costs)  of  using  the  least  expensive 
alternative  source  for  journal  articles  (if  the  center  were  not 
available)  are  summarized  as  giver,  in  Table  13.2. 

The  averag..  cost  to  professionals  foi  the  alternative  sources 
of  journal  article  readings  is  $75.60  per  reading  (including 
current  costs  plus  additional  cost  of  the  alternative).  This 
average  cost  of  alternative  sources  includes  the  following 
components  of  cost: 


Avg. 

Currant 

Coats 

Avg. 

CoMtof 

AKamatfvM 

Avg. 

Dllfsrence 

($) 

Professionala  time 

$34.20 

$64.00 

$30.70 

Time  of  others 

$2,20 

$1240 

$10,20 

Other  costs 

$0.80 

$1.10 

$0.30 

Total 


$37.20 


$7840 


$41C20 
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TABLE  13.2 

AMOUNT  OF  ADDITIONAL  TIME  IT  WOULD  TAKE  IN 
IDENTIFYING  LOCATING,  OBTAINING  AND 
PHOTOCOPYING  JOURNAL  ARTICLES  IF 
ORGANIZATION  PROFESSIONALS  DID  NOT  HAVE  AN 
INFORMATION  CENTER 


Activity 

Avg.  Time 

Par 

Reading 

(minutes) 

Total 

Amount 

Tf  Time  Par  Year 
(000  hours) 

Professional’s  Time 

Going  to  center 

28.3 

924 

Identity  article 

8.3 

27.1 

Locate  article 

63 

8.2 

Obtain  artlcla 

4.2 

134 

Photograph  article 

u. 

3.S 

Total 

48.2 

157.5 

Going  to  center 

9.1 

29.7 

Identify  article 

12.1 

395 

Locate  article 

5.4 

17.6 

Obtain  article 

4.2 

13.7 

Photograph  article 

U 

_3.S 

Total 

31.9 

104.2 

SOURCE:  King  Research,  Inc.  Survey  of  Professionals 


The  total  cost  of  alternative  sources  to  the  information 
center  is  about  S8.1  million  (i.c.,  $41.20  times  196,000 
readings  of  journal  articles  provided  by  the  center).  This 
amount  is  the  second  estimate  of  the  value  of  the 
information  center. 

A  third  way  to  look  at  the  value  of  the  center  in  providing 
access  to  journals  is  to  consider  that  professionals  seem  to 
spend  a  relatively  fixed  amount  of  time  in  information 
seeking  and  reading  (based  on  national  data  collected  by  us 
and  others  over  25  years).  If  we  assume  that  this  is  true  for 
professionals  in  this  example,  they  would  have  less  time  for 
this  activity  (i.e.,  obtaining  and  reading  articles)  if  the  center 
were  not  available  (or  did  not  exist).  The  total  time  the 
professionals  spend  identifying,  gaining  access  to  and 
reading  journal  articlc>  (accessed  though  the  center  service) 
is  175,500  hours.  The  additional  time  necessary  to  obtain 
articles  previously  provided  by  the  center  (if  the  services 
were  not  available)  is  about  157,500  hours,  thus  a  new 
average  time  per  article  read  is  1.7  hours  per  reading  (i.c , 
333,000  hours  divided  by  196,000  readings).  Dividing  this 
into  the  constant  hours  devoted  by  professionals  to  this 
activity  (175,500  hours)  yields  103,200  readings.  Therefore, 
about  92,800  readings  would  be  lost  to  professionals  (i.e., 
196,000  minus  103.200)  Assume  average  loss  in  savings 
attnbutable  to  reading  journal  articles  is  $450  or  11.8 
professional  hours  per  reading.  Total  loss  would  be  $842 
million  or  1.1  million  hours  of  professional  time.  Thus,  their 
productivity  would  be  affected.  In  addition  to  lost  savings  in 
time  the  lost  readings  would  also  have  some  effect  on 
quality,  timeliness  and  other  work  performance  factors  as 
well. 


13.4  The  Value  of  Books  Provided  by  the  Information 
Center 

Professionals  in  the  example  organization  have  520,000 
readings  from  books  per  year  of  which  124,000  of  these 
readings  are  from  books  provided  by  the  ceriter.  The 
approximate  amount  of  professional  time  spent  reading 
books  provided  by  the  center  is  estimated  to  be  about  1.7 
hours  spent  reading  and  0.35  hours  spent  identifying, 
locating  and  getting  the  books  (Table  13.3) 


TABLE  13.3 

AMOUNT  OF  1 IME  SPENT  IN  IDENTIFYING,  LOCATING, 
OBTAINING  AND  PHOTOCOPYING  BOOKS  OBTAINED 
FROM  THE  INFORMATION  CENTER  BY  ORGANIZATION 
PROFESSIONALS 


Activity 

Avg.  Time 

Per 

Reading 

(mkuites) 

Total 
Amount 
of  Time 
(000  hours) 

Professional’s  Time 

Going  to  center 

8.1 

13.7 

Identify  book 

3.6 

7.4 

Locate  book 

31 

6.4 

Obtain  book 

2.1 

43 

Photography  book 

SJL 

Total 

20.8 

43.0 

Someone  Else’?  Jlme 

Going  to  center 

4.0 

8.3 

Identify  book 

1.8 

37 

Locate  book 

1.2 

2.5 

Obtain  book 

1.1 

2.3 

Photography  book 

U 

U 

Total 

96 

203 

SOURCE:  Wng  Research,  Inc.  Survey  ol  Professionals 


The  total  time  spent  by  professionals  on  identifying  and 
accessing  the  center-provided  books  is  about  43,300  hours. 
The  total  time,  including  reading,  is  about  253,800  hours. 
The  amount  professionals  pay  in  terms  of  their  own  time 
getting  access  to  and  reading  books  is  about  $9.7  million 
0.e.,  the  minimum  value  to  them). 

About  75  percent  of  the  readings  of  books  provided  by  the 
center  involved  books  containing  information  whose 
existence  was  known  prior  to  reading.  About  72  percent  of 
the  time  the  respondents  indicated  the  information  could 
have  been  obtained  elsewhere,  such  as  from  an  external 
library  (30%),  a  colleague,  consultant,  etc.,  (26%),  they 
would  have  bought  it  (12%)  or  the  professionals’  own 
collection  (4%).  Even  though  professionals  know  about  the 
information  most  of  the  time,  the  cost  of  locating  and 
acquiring  it  is  expensive.  These  costs  are  summarized  as 
follows  in  Table  13.4. 
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The  ave  "»e  cost  per  reading  books  currently  obtained 
througii  the  center  and  by  using  alternatives  is  as  follows: 


Avg. 

Current 

Avg. 

Costs  of 

Avg 

Dlflerence 

IS) 

Prolessionals  time 

$78.40 

$103.50 

$25.10 

Tima  ol  others 

$  310 

$  1050 

$  7.40 

Other  costs 

$  15.60 

$  16  70 

$3.10 

Total 

$  97.10 

$132.70 

$3560 

Thus,  the  total  cost  of  alternatives  to  the  information  center 
IS  S4.4  million,  (i.e.,  124,000  times  S35.60). 


TABLE  13.4 

AMOUNT  OF  ADDITIONAL  TIME  IT  WOULD  TAKE  IN 
IDENTIFYING,  LOCATING,  OBTAINING  AND 
PHOTOCOPYING  BOOKS  IF  ORGANIZATION 
PROFESSIONALS  DID  NOT  HAVE  THE  INFORMATION 

Avg.  Time 

Total 

Per 

ArrxMKft 

Reeding 

of  Time  per  Year 

AaMiy 

(rnkurtes) 

(000  hours) 

pfglef^lpnfil'f  T'h® 

Going  10  center 

17.2 

355 

Identify  book 

5.8 

12.0 

locate  book 

6.4 

132 

Obtain  book 

52 

10.7 

Photograph  book 

A8 

9.9 

Total 

39  4 

61.4 

Going  to  center 

115 

238 

Identify  book 

29 

60 

Locate  book 

30 

62 

Obtain  book 

29 

60 

Photograph  book 

.£2 

Total 

23.2 

47.9 

SOURCE:  King  Research,  Inc.  Survey  ol  Prolessionals 


Finally,  the  additional  costs  to  professionals  (in  their  time) 
for  obtaining  the  book-related  information  are  about  8 1,400 
hoars,  or  a  new  total  of  about  338,200  hours  for  124,000 
readings.  Taking  this  additional  cost  into  account  results  in  a 
new  average  hours  per  reading  (2.7  hours  per  reading).  In 
order  to  maintain  a  constant  total  256,800  hours,  the 
number  of  readings  would  be  95,100  instead  of  124,000 
readings.  Therefore,  28,900  readings  would  be  lost.  The 
value  of  these  lest  readings  represents  the  potential  savings 
in  time  and  equipment  (i.e.,  $690  or  18.1  hours  of 
professional  time  per  reading)  as  well  as  improved  quality 
and  timeliness  of  work  that  would  have  been  achieved. 
Converted  to  professional  time,  this  comes  to  about  523,100 
hours.  The  total  value  calculated  in  this  manner  is  $20 
million  (i.e.,  28,900  readings  that  are  lost  times  $690  m 
savings  per  reading). 

13.5  The  Value  of  Internal  Documents  Provided  by  The 
Information  Center 

Professionals  read  about  360,000  internal  documents  of 
which  158,000  are  through  documents  obtained  at  the 
information  center.  The  professionals  are  estimated  to 
spend  an  average  of  about  1.0  hour  (per  reading)  in  reading 


internal  documents  and  0.31  hour  m  identifying,  locating 
and  getting  the  documents  to  read.  The  latter  estimates  arc 
subdivided  as  shown  in  Table  13.5. 

TABLE  13.5 

AMOUNT  OF  TIME  SPENT  IN  IDENTIFYING,  LOCATING, 
OBTAINING  AND  PHOTOCOPYING  INTERNAL 
DOCUMENTS  OBTAINED  FROM  THE  INFORMATION 
CENTER  BY  ORGANIZATION  PROFESSIONALS 


Activity 

Avg.  Time 

Per 

Reeding 

(minutes) 

Total 

Amount 

of  Time  per  Year 
(000  hours) 

Going  to  center 

43 

113 

Identify  Interna)  doctimenis 

44 

116 

Locate  internal  documents 

1.5 

40 

Obtain  internal  documents 

7.9 

208 

Photograph  internal  documents 

JIZ 

-tS 

Total 

186 

49  5 

Going  to  center 

74 

identify  interna)  documents 

1  6 

42 

Locate  interna)  documents 

1.7 

45 

Obtain  Intarnai  documants 

20 

53 

Photograph  interna)  documents 

-as 

JW 

Total 

90 

238 

SOURCE’  Wng  Research,  Inc  Survey  ol  Professionals 

The  estimated  total  time  spent  by  professionals  in 
identifying,  locating,  obtaining  and  reading  documents 
provided  by  the  center  is  207,000  hours  (i.e.,  158,000 
readings  times  1.31  hours  per  reading).  Thus,  the 
■Villingness  to  p.iy"  value  is  $7,9  million. 

The  information  found  in  rend  internal  documents  was 
known  by  the  readers  for  53  percent  of  the  readings.  They 
indicated  that  some  would  not  have  obtained  the  document 
01  information,  if  the  center  was  not  available  Most 
of  them  would  get  it  from  a  colleague  (47%),  an  alternate 
library  (23%)  or  elsewhere  (5%). 

The  average  additional  costs  of  obtaining  internal 
documents  or  information  found  in  them  from  alternative 
sources  to  the  center  are  given  in  Table  13.6. 


The  average  cost  for  using  alternative  sources  is  $12.00  per 
reading  as  shown  below. 


Avg. 

Current 

Costs 

Avg. 

Costs  of 
Alternatives 

Avg 

Difference 

($) 

Professionals  time 

$50.10 

$6030 

$1020 

Time  of  others 

$2.90 

$4.20 

$  1.30 

Othar  coats 

$090 

$  1.40 

$050 

Tolal 

$5390 

$65  90 

$12.00 
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The  total  cost  of  using  alternatives  for  the  158  f  00  readings 
of  internal  documents  is  $  1 .9  million. 

A  total  of  42,000  hours  of  additional  professionals’  time 
would  be  involved  in  using  alternative  sources.  Thus,  a  total 
of  249,000  hours  would  be  required  to  obtain  and  read 
158,000  documents  —  or  1.58  hours  per  reading.  If  the 
professionals  continue  to  spend  207,000  hours  with  these 
documents,  they  would  be  able  to  read  131,000  at  1.58 
hours  per  reading.  This  means  they  would  lose  about  27,(;(*'  > 
readings.  Savings  for  these  27,000  readings  is  estimated  to 
be  $1,210  a  piece  (on  the  average)  or  a  total  of  $33  million. 
In  terms  of  professionals’  time  this  value  is  850,000  hours. 

TABLE  13.6 

AMOUNT  OF  ADDITIONAL  TIME  IT  WOULD  TAKE  IN 
IDENTIFYING  LOCATING,  OBTAINING  AND 
PHOTOCOPYING  INTERNAL  DOCUMENTS  IF 
ORGANIZATION  PROFESSIONALS  DID  NOT  HAVE  THE 
INFORMATION  CENTER 


ActMty 

Avg.  Time 

Per 

Reading 

{mlnutaa) 

Total 

Amount 

of  Time  per  Year 
(000  hous) 

PWfVMtoIrt’l  TllM 

Going  to  c«nt«r 

7.0 

184 

tdontify  (nt«mid  documents 

1.6 

4.2 

locate  internal  documents 

34 

90 

Obtain  internal  documents 

26 

68 

Photograph  internal  documents 

04 

Total 

160 

42.1 

Going  to  e*nl«r 


identify  internal  documents 

1  2 

32 

Locate  Internal  documents 

1.2 

32 

Obtain  internal  documents 

1.2 

32 

Photograph  internal  documents 

.s,s 

Total 

41 

109 

SOURCE.  Wng  Rosovet).  (no  Survey  ol  Professionels 

13.6  Value  of  Other  Services 

The  value  of  searches  performed  by  the  information  center 
staff  ir  hew  much  more  it  would  cost  to  do  the  searches  if 
theie  were  no  center  staff  available  to  search.  We  find  that 
about  19  percent  of  the  searches  would  not  have  been  done. 
Over  four-fifths  of  the  searches  would  have  been  delegated 
to  someone  else  on  the  professional’s  staff;  1 2  percent  would 
have  been  obtained  from  an  external  library;  and  20  percent, 
called  a  knowledgeable  colleague  or  used  a  contractor  or 
online  service.  The  cost  of  using  alternative  means  of 
searching  is  estimated  to  be  about  $240  more  than  is 
currently  spent  on  the  searches  (about  $110).  Most  of  this 
additional  cost  is  in  terms  of  the  users’  time  fS59).  but  some 
of  it  is  in  the  additional  cost  to  purchase  a  search  ($21), 
someone  else’s  time  ($8)  or  other  expense  ($22).  Thus  all 


told.it  would  cost  the  organization  about  $1.1  million  more 
to  conduct  the  searches  without  the  benefit  of  the  center 
staff. 

The  users  of  Current  Awareness  Bulletins  indicated  a 
number  of  ways  in  which  they  benefitted  by  having  the  last 
Current  Awareness  Bulletin  as  follows: 

•  Identified  needed  sources  that  they  probably  would 

not  have  identified  otherwise  58% 

•  Identified  needed  sources  sooner  than  they  could  have 

otherwise  29% 

•  Saved  them  or  their  staff  time  in  identifying  needed 

documents  21% 

They  indicated  it  would  require  an  average  of  about  3.4 

hours  of  their  time  or  ,iicir  staffs’  time  to  identify  needed 
documents  themselves. 


If  the  Current  Awareness  Bulletin  was  not  provided  to  the 
professionals,  the  users  would  have  identifled  the  needed 
sources  m  the  following  manner: 


They  could  not  have  done  it 

59% 

Department  circulation/routing 

17% 

They  would  have  conducted  an  online  search 

9% 

They  would  have  delegated  an  online  record 

8% 

Other  means 

8% 

The  cost  of  using  the  other  source  is  estimated  to  be  about 
$57  per  use  of  the  Bulletins.  This  cost  of  alternative  sources 
IS  derived  from: 

• 

Using  their  own  time 

$34.00 

# 

Using  the  time  of  others 

$12,00 

• 

Cost  of  purchasing  a  search  service 

$10.60 

• 

Telephone  calls  and  other 

S  0.30 

13.7  Total  Value  of  The  Information  Center 


As  indicated  above,  the  value  of  the  information  center 
services  can  be  assessed  from  thre,  perspectives:  what  users 
are  willing  to  pay  (in  terms  of  their  time  and  effort)  for 
information  provided  by  the  center,  what  it  would  cost  them 
to  use  alternative  sourers  for  obtaining  the  information,  and 
what  savings  (or  research  cost  avoidance)  would  be  lost  if  the 
center  did  not  exist.  Rough  estimates  for  these  three 
perspectives  of  value  are  given  below. 

The  return-on-information  of  this  cost  is  substantial, 
regardless  of  how  one  views  value.  That  is,  the  return  is: 

•  4.3  to  one  in  terms  of  willingness  to  pay, 

•  2,5  to  one  in  terms  of  cost  to  use  alternative  sources, 
and 

•  15  to  one  in  terms  of  research  cost  avoidance  (savings) 
These  returns  are  impressive  indeed. 
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TABLE  13.7 

THE  VALUE  OF  INFORMATION  CENTER  SERVICES 
FROM  THREE  PERSPECTIVES 
AND  BY  SOURCE  OF  READING 


Willingness 
to  Pay 

Cost  to  Use 
Alternative 
Sources 

Cost 

Avoidance 
by  Savings 

Journals 

$  7.3  million 

$8.1  million 

$42  million 

Books 

$  9.7  million 

$4.4  million 

$20  million 

Internal  documents 

$  7.9  million 

$1.9  million 

$33  million 

Online  searching 

$  1.C  million 

$1.1  million 

_ 

Current  Awareness 
Bulletins 

$  0.2  million 

$0.7  million 

_ 

Total 

$26.6  million 

$15.3  million 

$  95  million 

SOURCE:  King  Research,  Inc. 
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